* Re: [dpdk-dev] [PATCH] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
@ 2021-03-05 4:05 ` Ajit Khaparde
2021-03-10 11:11 ` Min Hu (Connor)
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
` (11 subsequent siblings)
12 siblings, 1 reply; 80+ messages in thread
From: Ajit Khaparde @ 2021-03-05 4:05 UTC (permalink / raw)
To: Lijun Ou; +Cc: Ferruh Yigit, dpdk-dev, linuxarm
[-- Attachment #1: Type: text/plain, Size: 795 bytes --]
On Thu, Mar 4, 2021 at 5:03 PM Lijun Ou <oulijun@huawei.com> wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./testpmd -w xxx --file-prefix=xx -l 0-1 -n 2 -- -i\
> --rxq=16 --txq=16 --num-procs=2 --proc-id=0
> the secondary cmd:
> ./testpmd -w xxx --file-prefix=xx -l 2-3 -n 2 -- -i\
> --rxq=16 --txq=16 --num-procs=2 --proc-id=1
>
Can you update this in the testpmd usage guide.
There are so many things that testpmd can do.
What will happen to the stats if I quit one of the processes and start
again?
In case of multiple Rx queues, how will RSS happen?
It will be good to have the behavior of various features documented -
some of the questions have been already answered.
They can be documented as well.
--
> 2.7.4
>
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH] app/testpmd: support multi-process
2021-03-05 4:05 ` Ajit Khaparde
@ 2021-03-10 11:11 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-10 11:11 UTC (permalink / raw)
To: dev
在 2021/3/5 12:05, Ajit Khaparde 写道:
> On Thu, Mar 4, 2021 at 5:03 PM Lijun Ou <oulijun@huawei.com> wrote:
>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./testpmd -w xxx --file-prefix=xx -l 0-1 -n 2 -- -i\
>> --rxq=16 --txq=16 --num-procs=2 --proc-id=0
>> the secondary cmd:
>> ./testpmd -w xxx --file-prefix=xx -l 2-3 -n 2 -- -i\
>> --rxq=16 --txq=16 --num-procs=2 --proc-id=1
>>
> Can you update this in the testpmd usage guide.
OK, I will do it in V2.
> There are so many things that testpmd can do.
> What will happen to the stats if I quit one of the processes and start
> again?
stats will not change when one quit and start, As they share the same
buffer to store the stats.
> In case of multiple Rx queues, how will RSS happen?
Primary process and secondary process has seperate queues to use, RSS
will work in their own queues whether primary and secondary process.
> It will be good to have the behavior of various features documented -
> some of the questions have been already answered.
> They can be documented as well.
OK, I will do it in V2, thanks.
>
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v2] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
2021-03-05 4:05 ` Ajit Khaparde
@ 2021-03-11 2:47 ` Min Hu (Connor)
2021-03-22 2:27 ` Ajit Khaparde
` (4 more replies)
2021-03-11 9:07 ` [dpdk-dev] [PATCH v3] " Min Hu (Connor)
` (10 subsequent siblings)
12 siblings, 5 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-11 2:47 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="jing?6345", Size: 15396 bytes --]
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
app/test-pmd/cmdline.c | 12 ++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
6 files changed, 196 insertions(+), 50 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 576d5ac..9edf84a 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c8acd5d..1a5bf25 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -605,6 +607,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1366,6 +1370,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1a57324..bbd45d9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,8 @@
#include "testpmd.h"
+int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -982,16 +987,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
/* wrapper to rte_mempool_create() */
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size, socket_id);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, socket_id);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_ANON:
{
- rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf,
- mb_size, (unsigned int) mb_mempool_cache,
- sizeof(struct rte_pktmbuf_pool_private),
- socket_id, mempool_flags);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_mempool_create_empty(pool_name,
+ nb_mbuf, mb_size,
+ (unsigned int)mb_mempool_cache,
+ sizeof(struct rte_pktmbuf_pool_private),
+ socket_id, mempool_flags);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
if (rte_mp == NULL)
goto err;
@@ -1021,9 +1036,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size,
- heap_socket);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, heap_socket);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_XBUF:
@@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2503,21 +2528,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2618,15 +2650,18 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- if (rte_eth_dev_start(pi) < 0) {
- printf("Fail to start port %d\n", pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d\n", pi);
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to "
"stopped\n", pi);
- continue;
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2755,7 +2790,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2824,8 +2860,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3089,7 +3127,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3611,6 +3649,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3709,13 +3751,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3746,12 +3781,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3776,10 +3815,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3875,8 +3910,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3922,6 +3959,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3957,6 +3996,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index ce83f31..6d13417 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+uint8_t f_quit;
+int testpmd_fd_copy;
+struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -422,6 +427,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 6745072..7e1b95d 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -536,3 +536,72 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+----------------------------
+
+The following are the command-line options for the testpmd applications(support multi process).
+They must be separated from the EAL options, shown in the previous section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the same in primary process
+ and secondary process.
+
+* ``--proc-type``
+ Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
+
+* ``-l CORELIST``
+ List of cores to run on. the corelist should be different in primary process and secondary process.
+
+* ``--rxq=N``
+
+ Set the number of RX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--txq=N``
+
+ Set the number of TX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+Most dev ops is supported in primary and secondary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows:
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
+RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v2] app/testpmd: support multi-process
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
@ 2021-03-22 2:27 ` Ajit Khaparde
2021-03-22 6:35 ` Min Hu (Connor)
2021-06-15 12:23 ` [dpdk-dev] [PATCH v14] " Min Hu (Connor)
` (3 subsequent siblings)
4 siblings, 1 reply; 80+ messages in thread
From: Ajit Khaparde @ 2021-03-22 2:27 UTC (permalink / raw)
To: Min Hu (Connor); +Cc: dpdk-dev, Ferruh Yigit
[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]
:::snip::::
> +* ``--proc-type``
> + Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
> +
> +* ``-l CORELIST``
> + List of cores to run on. the corelist should be different in primary process and secondary process.
> +
> +* ``--rxq=N``
> +
> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary process.
If RxQ is 1, how will it be shared between the primary and secondary processes?
Should the number of rings be a multiple of the number of processes?
> +
> +* ``--txq=N``
> +
> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary process.
Same as above. How will a single ring be used across primary and
secondary processes?
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary process and secondary process.
> +
> +Calculation rule for queue:
> +All queues are allocated to different processes based on proc_num and proc_id.
> +Calculation rule for the Testpmd to allocate queues to each process:
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
> +For example, if supports 4 txq and rxq
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +Most dev ops is supported in primary and secondary process. While secondary process is not permitted
> +to allocate or release shared memory, so some ops are not supported as follows:
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
> +stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
> +RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v2] app/testpmd: support multi-process
2021-03-22 2:27 ` Ajit Khaparde
@ 2021-03-22 6:35 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-22 6:35 UTC (permalink / raw)
To: Ajit Khaparde; +Cc: dpdk-dev, Ferruh Yigit
在 2021/3/22 10:27, Ajit Khaparde 写道:
> :::snip::::
>
>> +* ``--proc-type``
>> + Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
>> +
>> +* ``-l CORELIST``
>> + List of cores to run on. the corelist should be different in primary process and secondary process.
>> +
>> +* ``--rxq=N``
>> +
>> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
>> + The default value is 1. N is the sum of queues used by primary and secondary process.
> If RxQ is 1, how will it be shared between the primary and secondary processes?
Yes, your are right.the range of RxQ(or TxQ) is wrong.
As primary instance and secondary instance should have separate queues,
and each should occupy at least one queue. So the range of RxQ(or TxQ)
shoud be no less than two.
> Should the number of rings be a multiple of the number of process The number of rings had better be a multiple of the number of processes.
If not, redundant queues will exist after queues are allocated to
processes. After RSS is enabled, packet loss occurs when traffic is sent
to all processes at the same time. Some traffic enters redundant queues
and cannot be forwarded.
I will fix it in v4.Thanks.
>> +
>> +* ``--txq=N``
>> +
>> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
>> + The default value is 1. N is the sum of queues used by primary and secondary process.
> Same as above. How will a single ring be used across primary and
> secondary processes?
Just same as above, I will fix it in v4.
Thanks.
>
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
>> +* ``--proc-id=id``
>> +
>> + The id of the current process (id < num-procs). id should be different in primary process and secondary process.
>> +
>> +Calculation rule for queue:
>> +All queues are allocated to different processes based on proc_num and proc_id.
>> +Calculation rule for the Testpmd to allocate queues to each process:
>> +start(queue start id) = proc_id * nb_q / num_procs；
>> +end(queue end id) = start + nb_q / num_procs；
>> +
>> +For example, if supports 4 txq and rxq
>> +the 0~1 for primary process
>> +the 2~3 for secondary process
>> +
>> +Most dev ops is supported in primary and secondary process. While secondary process is not permitted
>> +to allocate or release shared memory, so some ops are not supported as follows:
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
>> +
>> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
>> +stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
>> +RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
>> --
>> 2.7.4
>>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v14] app/testpmd: support multi-process
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
2021-03-22 2:27 ` Ajit Khaparde
@ 2021-06-15 12:23 ` Min Hu (Connor)
2021-07-02 12:09 ` [dpdk-dev] [PATCH v15] " Andrew Rybchenko
` (2 subsequent siblings)
4 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-06-15 12:23 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, andrew.rybchenko
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v14:
* Fixed comments by Andrew Rybchenko.
v13:
* Modified the doc syntax.
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 20 +++++-
app/test-pmd/parameters.c | 9 +++
app/test-pmd/testpmd.c | 121 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_08.rst | 2 +-
doc/guides/testpmd_app_ug/run_app.rst | 70 +++++++++++++++++++
7 files changed, 212 insertions(+), 25 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 0268b18..a215b12 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5463,6 +5463,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush Rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 43c79b5..a0c24bb 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2976,6 +2976,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2993,7 +2995,21 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs != 0)
+ printf("Warning! queue numbers should be multiple of processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -3010,6 +3026,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..0f09841 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,10 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (!strcmp(lgopts[opt_idx].name, PARAM_NUM_PROCS))
+ num_procs = atoi(optarg);
+ if (!strcmp(lgopts[opt_idx].name, PARAM_PROC_ID))
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 8ed1b97..6eaf8f8 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -515,6 +515,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * ID of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -974,6 +1030,15 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ if (rte_mp == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ return rte_mp;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1995,6 +2060,11 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd Rx queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2482,21 +2552,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2519,8 +2592,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2597,16 +2670,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
continue;
}
@@ -2731,7 +2804,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2800,8 +2873,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3067,7 +3142,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3398,7 +3473,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3588,6 +3663,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3754,10 +3833,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 283b5e3..76e10b5 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -631,6 +631,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index a6ecfdf..4b1ea4c 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -54,7 +54,7 @@ New Features
This section is a comment. Do not overwrite or remove it.
Also, make sure to start the actual text at the margin.
=======================================================
-
+ * Added support multi-process for testpmd.
Removed Items
-------------
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index eb48318..4c13a23 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -545,3 +545,73 @@ The command line options are:
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in
+ primary process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if testpmd supports 4 Tx and Rx queues
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS
+is enabled, packet loss occurs when traffic is sent to all processes at the same
+time. Some traffic enters redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is
+not permitted to allocate or release shared memory, so some ops are not supported
+as follows::
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in
+secondary process, like::
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+Stats is supported, stats will not change when one quit and start, as they
+share the same buffer to store the stats. Flow rules are maintained in process
+level: primary and secondary has its own flow list (but one flow list in HW).
+The two can see all the queues, so setting the flow rules for the other is OK.
+But in the testpmd primary process receiving or transmitting packets from the
+queue allocated for secondary process is not permitted, and same for secondary
+process.
+
+Flow API and RSS are supported.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
2021-03-22 2:27 ` Ajit Khaparde
2021-06-15 12:23 ` [dpdk-dev] [PATCH v14] " Min Hu (Connor)
@ 2021-07-02 12:09 ` Andrew Rybchenko
2021-07-02 12:47 ` Andrew Rybchenko
2021-07-10 3:50 ` [dpdk-dev] [PATCH v16] " Min Hu (Connor)
2021-08-25 2:06 ` [dpdk-dev] [PATCH v17] " Min Hu (Connor)
4 siblings, 1 reply; 80+ messages in thread
From: Andrew Rybchenko @ 2021-07-02 12:09 UTC (permalink / raw)
To: Xiaoyun Li, Anatoly Burakov
Cc: dev, Min Hu (Connor), Lijun Ou, Ajit Khaparde, Ferruh Yigit
From: "Min Hu (Connor)" <humin29@huawei.com>
For example the following commands run two testpmd processes:
* the primary process:
./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
* the secondary process:
./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v15:
* Fixed release notes.
* Cleanup documentation.
v14:
* Fixed comments by Andrew Rybchenko.
v13:
* Modified the doc syntax.
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 20 +++-
app/test-pmd/parameters.c | 9 ++
app/test-pmd/testpmd.c | 121 ++++++++++++++++++++-----
app/test-pmd/testpmd.h | 9 ++
doc/guides/rel_notes/release_21_08.rst | 5 +
doc/guides/testpmd_app_ug/run_app.rst | 82 +++++++++++++++++
7 files changed, 228 insertions(+), 24 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 8468018cf..5b9880dde 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5459,6 +5459,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush Rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 04ae0feb5..9e4d86b4a 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2976,6 +2976,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2993,7 +2995,21 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs != 0)
+ printf("Warning! queue numbers should be multiple of processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -3010,6 +3026,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 5e69d2aa8..925fca562 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -509,6 +509,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -628,6 +631,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1394,6 +1399,10 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (!strcmp(lgopts[opt_idx].name, PARAM_NUM_PROCS))
+ num_procs = atoi(optarg);
+ if (!strcmp(lgopts[opt_idx].name, PARAM_PROC_ID))
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1cdd3cdd1..a5da0c272 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * ID of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -983,6 +1039,15 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
#endif
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ if (rte_mp == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ return rte_mp;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -2008,6 +2073,11 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd Rx queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2495,21 +2565,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2532,8 +2605,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2610,16 +2683,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d cannot be set back to stopped\n",
+ pi);
continue;
}
@@ -2744,7 +2817,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2813,8 +2886,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3082,7 +3157,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3413,7 +3488,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3603,6 +3678,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3771,10 +3850,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index d61a055bd..11b9451b0 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index a6ecfdf3c..8ba0bfb83 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -55,6 +55,11 @@ New Features
Also, make sure to start the actual text at the margin.
=======================================================
+ * **Added multi-process support to testpmd.**
+
+ Added command-line options to specify total number of processes and
+ current process ID. Each process owns subset of Rx and Tx queues.
+
Removed Items
-------------
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index eb4831835..348e5fcac 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -545,3 +545,85 @@ The command line options are:
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+* primary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+* secondary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=ID``
+
+ The ID of the current process (ID < num-procs). ID should be different in
+ primary process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and
+``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+
+* start(queue start id) = proc_id * nb_q / num_procs
+
+* end(queue end id) = start + nb_q / num_procs
+
+For example, if testpmd is configured to have 4 Tx and Rx queues,
+queues 0 and 1 will be used by the primary process and
+queues 2 and 3 will be used by the secondary process.
+
+The number of queues should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. If RSS
+is enabled, packet loss occurs when traffic is sent to all processes at the same
+time. Some traffic goes to redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is
+not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+
+- ``dev_configure``
+- ``dev_start``
+- ``dev_stop``
+- ``rx_queue_setup``
+- ``tx_queue_setup``
+- ``rx_queue_release``
+- ``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in
+secondary process, like:
+
+.. code-block:: console
+
+ port config all rxq|txq|rxd|txd <value>
+ port config <port_id> rx_offload xxx on/off
+ port config <port_id> tx_offload xxx on/off
+
+etc.
+
+Stats is supported, stats will not change when one quits and starts, as they
+share the same buffer to store the stats. Flow rules are maintained in process
+level: primary and secondary has its own flow list (but one flow list in HW).
+The two can see all the queues, so setting the flow rules for the other is OK.
+But in the testpmd primary process receiving or transmitting packets from the
+queue allocated for secondary process is not permitted, and same for secondary
+process.
+
+Flow API and RSS are supported.
--
2.30.2
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
2021-07-02 12:09 ` [dpdk-dev] [PATCH v15] " Andrew Rybchenko
@ 2021-07-02 12:47 ` Andrew Rybchenko
2021-07-08 12:20 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Andrew Rybchenko @ 2021-07-02 12:47 UTC (permalink / raw)
To: Xiaoyun Li, Anatoly Burakov
Cc: dev, Min Hu (Connor), Lijun Ou, Ajit Khaparde, Ferruh Yigit
On 7/2/21 3:09 PM, Andrew Rybchenko wrote:
> From: "Min Hu (Connor)" <humin29@huawei.com>
>
> For example the following commands run two testpmd processes:
>
> * the primary process:
>
> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> * the secondary process:
>
> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
[snip]
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 1cdd3cdd1..a5da0c272 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
> */
> uint32_t eth_link_speed;
>
> +/*
> + * ID of the current process in multi-process, used to
> + * configure the queues to be polled.
> + */
> +int proc_id;
> +
> +/*
> + * Number of processes in multi-process, used to
> + * configure the queues to be polled.
> + */
> +unsigned int num_procs = 1;
> +
> +static int
> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
> + const struct rte_eth_conf *dev_conf)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
> + dev_conf);
> + return 0;
> +}
> +
> +static int
> +eth_dev_start_mp(uint16_t port_id)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_start(port_id);
> +
> + return 0;
> +}
> +
> +static int
> +eth_dev_stop_mp(uint16_t port_id)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_stop(port_id);
> +
> + return 0;
> +}
> +
> +static void
> +mempool_free_mp(struct rte_mempool *mp)
> +{
> + if (is_proc_primary())
> + rte_mempool_free(mp);
> +}
> +
> +static int
> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_set_mtu(port_id, mtu);
> +
> + return 0;
> +}
> +
I think above functions should be removed and corresponding
checks should be done in caller directly since above functions
are used in single place only and just hide what actually
happens in the case of secondary process. It is very
misleading.
[snip]
> @@ -2495,21 +2565,24 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> - nb_txq + nb_hairpinq,
> - &(port->dev_conf));
> + diag = eth_dev_configure_mp(pi,
> + nb_rxq + nb_hairpinq,
> + nb_txq + nb_hairpinq,
> + &(port->dev_conf));
> if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d cannot be set back to stopped\n",
> + pi);
Unrelated changes in the patch should be avoided since
it just makes the review harder.
> printf("Fail to configure port %d\n", pi);
> /* try to reconfigure port next time */
> port->need_reconfig = 1;
> return -1;
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2532,8 +2605,8 @@ start_port(portid_t pid)
> if (rte_atomic16_cmpset(&(port->port_status),
> RTE_PORT_HANDLING,
> RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> + printf("Port %d cannot be set back to stopped\n",
> + pi);
Unrelated changes in the patch should be avoided.
> printf("Fail to configure port %d tx queues\n",
> pi);
> /* try to reconfigure queues next time */
> @@ -2610,16 +2683,16 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> + diag = eth_dev_start_mp(pi);
> if (diag < 0) {
> printf("Fail to start port %d: %s\n", pi,
> rte_strerror(-diag));
>
> /* Fail to setup rx queue, return */
> if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> - "stopped\n", pi);
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + printf("Port %d cannot be set back to stopped\n",
> + pi);
Unrelated changes in the patch should be avoided.
[snip]
> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
> index eb4831835..348e5fcac 100644
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -545,3 +545,85 @@ The command line options are:
> bit 0 - two hairpin ports loop
>
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +
> +Testpmd Multi-Process Command-line Options
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The following are the command-line options for testpmd multi-process support:
> +
> +* primary process:
> +
> +.. code-block:: console
> +
> + sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=0
> +
> +* secondary process:
> +
> +.. code-block:: console
> +
> + sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=1
> +
> +The command line options are:
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=ID``
> +
> + The ID of the current process (ID < num-procs). ID should be different in
> + primary process and secondary process, which starts from '0'.
> +
> +Calculation rule for queue:
> +All queues are allocated to different processes based on ``proc_num`` and
> +``proc_id``.
> +Calculation rule for the testpmd to allocate queues to each process:
> +
> +* start(queue start id) = proc_id * nb_q / num_procs
> +
> +* end(queue end id) = start + nb_q / num_procs
> +
> +For example, if testpmd is configured to have 4 Tx and Rx queues,
> +queues 0 and 1 will be used by the primary process and
> +queues 2 and 3 will be used by the secondary process.
> +
> +The number of queues should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. If RSS
> +is enabled, packet loss occurs when traffic is sent to all processes at the same
> +time. Some traffic goes to redundant queues and cannot be forwarded.
> +
> +All the dev ops is supported in primary process. While secondary process is
> +not permitted to allocate or release shared memory, so some ops are not supported
> +as follows:
> +
> +- ``dev_configure``
> +- ``dev_start``
> +- ``dev_stop``
> +- ``rx_queue_setup``
> +- ``tx_queue_setup``
> +- ``rx_queue_release``
> +- ``tx_queue_release``
> +
> +So, any command from testpmd which calls those APIs will not be supported in
> +secondary process, like:
> +
> +.. code-block:: console
> +
> + port config all rxq|txq|rxd|txd <value>
> + port config <port_id> rx_offload xxx on/off
> + port config <port_id> tx_offload xxx on/off
> +
> +etc.
I did the formatting cleanup, but I still think that testpmd
guide should not dive into such level of details. It should
rather highlight multi-process behaviour specifics.
Shouldn't testpmd store state in shared memory to avoid
problems when primary is stopped while secondary is running
etc.
Some testpmd features rely on reconfigure (i.e. simply change
configuration and set flag that reconfigure is required), but
configure does nothing and will simply ignore new settings.
So, it could look very-very confusing from user point of view.
I'm not sure that it is acceptable to apply the patch in such
state and open huge number of bugs in testpmd behaviour when
multi-process is used.
I'd even consider to exclude unsupported commands from help
etc. However, such level of care about user could be excessive
for test tool.
IMHO, it should be no requirement to repeat the primary
process command-line configuration in the second process
command line (see --rxq=4 --txq=4 above). The information
should be obtained from shared state. In theory primary
process could even change some settings in interactive
mode. I think testpmd should guarantee consistent behaviour
even in such conditions. I.e. do not allow to stop ports
used by forwarding running in secondary processes.
Run-time queues setup and deferred start should be very
carefully handled as well.
> +
> +Stats is supported, stats will not change when one quits and starts, as they
> +share the same buffer to store the stats. Flow rules are maintained in process
> +level: primary and secondary has its own flow list (but one flow list in HW).
> +The two can see all the queues, so setting the flow rules for the other is OK.
> +But in the testpmd primary process receiving or transmitting packets from the
> +queue allocated for secondary process is not permitted, and same for secondary
> +process.
> +
> +Flow API and RSS are supported.
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
2021-07-02 12:47 ` Andrew Rybchenko
@ 2021-07-08 12:20 ` Min Hu (Connor)
2021-07-08 12:30 ` Andrew Rybchenko
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-07-08 12:20 UTC (permalink / raw)
To: Andrew Rybchenko, Xiaoyun Li, Anatoly Burakov
Cc: dev, Lijun Ou, Ajit Khaparde, Ferruh Yigit
Hi, Andrew ,
在 2021/7/2 20:47, Andrew Rybchenko 写道:
> On 7/2/21 3:09 PM, Andrew Rybchenko wrote:
>> From: "Min Hu (Connor)" <humin29@huawei.com>
>>
>> For example the following commands run two testpmd processes:
>>
>> * the primary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> * the secondary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> [snip]
>
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index 1cdd3cdd1..a5da0c272 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
>> */
>> uint32_t eth_link_speed;
>>
>> +/*
>> + * ID of the current process in multi-process, used to
>> + * configure the queues to be polled.
>> + */
>> +int proc_id;
>> +
>> +/*
>> + * Number of processes in multi-process, used to
>> + * configure the queues to be polled.
>> + */
>> +unsigned int num_procs = 1;
>> +
>> +static int
>> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
>> + const struct rte_eth_conf *dev_conf)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
>> + dev_conf);
>> + return 0;
>> +}
>> +
>> +static int
>> +eth_dev_start_mp(uint16_t port_id)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_start(port_id);
>> +
>> + return 0;
>> +}
>> +
>> +static int
>> +eth_dev_stop_mp(uint16_t port_id)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_stop(port_id);
>> +
>> + return 0;
>> +}
>> +
>> +static void
>> +mempool_free_mp(struct rte_mempool *mp)
>> +{
>> + if (is_proc_primary())
>> + rte_mempool_free(mp);
>> +}
>> +
>> +static int
>> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_set_mtu(port_id, mtu);
>> +
>> + return 0;
>> +}
>> +
>
> I think above functions should be removed and corresponding
> checks should be done in caller directly since above functions
> are used in single place only and just hide what actually
> happens in the case of secondary process. It is very
> misleading.
>
This was done as Ferruh suggested in V9， and this could reduce
the complexity for testpmd when added by the multi-process support.
> [snip]
>
>> @@ -2495,21 +2565,24 @@ start_port(portid_t pid)
>> return -1;
>> }
>> /* configure port */
>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>> - nb_txq + nb_hairpinq,
>> - &(port->dev_conf));
>> + diag = eth_dev_configure_mp(pi,
>> + nb_rxq + nb_hairpinq,
>> + nb_txq + nb_hairpinq,
>> + &(port->dev_conf));
>> if (diag != 0) {
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> + if (rte_atomic16_cmpset(
>> + &(port->port_status),
>> + RTE_PORT_HANDLING,
>> + RTE_PORT_STOPPED) == 0)
>> + printf("Port %d cannot be set back to stopped\n",
>> + pi);
>
> Unrelated changes in the patch should be avoided since
> it just makes the review harder.This will be fixed in v16.
>
>> printf("Fail to configure port %d\n", pi);
>> /* try to reconfigure port next time */
>> port->need_reconfig = 1;
>> return -1;
>> }
>> }
>> - if (port->need_reconfig_queues > 0) {
>> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
>> port->need_reconfig_queues = 0;
>> /* setup tx queues */
>> for (qi = 0; qi < nb_txq; qi++) {
>> @@ -2532,8 +2605,8 @@ start_port(portid_t pid)
>> if (rte_atomic16_cmpset(&(port->port_status),
>> RTE_PORT_HANDLING,
>> RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> + printf("Port %d cannot be set back to stopped\n",
>> + pi);
>
> Unrelated changes in the patch should be avoided.
This will be fixed in v16.
>
>> printf("Fail to configure port %d tx queues\n",
>> pi);
>> /* try to reconfigure queues next time */
>> @@ -2610,16 +2683,16 @@ start_port(portid_t pid)
>> cnt_pi++;
>>
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> + diag = eth_dev_start_mp(pi);
>> if (diag < 0) {
>> printf("Fail to start port %d: %s\n", pi,
>> rte_strerror(-diag));
>>
>> /* Fail to setup rx queue, return */
>> if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back to "
>> - "stopped\n", pi);
>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> + printf("Port %d cannot be set back to stopped\n",
>> + pi);
>
> Unrelated changes in the patch should be avoided.
This will be fixed in v16.
>
> [snip]
>
>> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
>> index eb4831835..348e5fcac 100644
>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>> @@ -545,3 +545,85 @@ The command line options are:
>> bit 0 - two hairpin ports loop
>>
>> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
>> +
>> +
>> +Testpmd Multi-Process Command-line Options
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The following are the command-line options for testpmd multi-process support:
>> +
>> +* primary process:
>> +
>> +.. code-block:: console
>> +
>> + sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
>> + --num-procs=2 --proc-id=0
>> +
>> +* secondary process:
>> +
>> +.. code-block:: console
>> +
>> + sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
>> + --num-procs=2 --proc-id=1
>> +
>> +The command line options are:
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
>> +* ``--proc-id=ID``
>> +
>> + The ID of the current process (ID < num-procs). ID should be different in
>> + primary process and secondary process, which starts from '0'.
>> +
>> +Calculation rule for queue:
>> +All queues are allocated to different processes based on ``proc_num`` and
>> +``proc_id``.
>> +Calculation rule for the testpmd to allocate queues to each process:
>> +
>> +* start(queue start id) = proc_id * nb_q / num_procs
>> +
>> +* end(queue end id) = start + nb_q / num_procs
>> +
>> +For example, if testpmd is configured to have 4 Tx and Rx queues,
>> +queues 0 and 1 will be used by the primary process and
>> +queues 2 and 3 will be used by the secondary process.
>> +
>> +The number of queues should be a multiple of the number of processes. If not,
>> +redundant queues will exist after queues are allocated to processes. If RSS
>> +is enabled, packet loss occurs when traffic is sent to all processes at the same
>> +time. Some traffic goes to redundant queues and cannot be forwarded.
>> +
>> +All the dev ops is supported in primary process. While secondary process is
>> +not permitted to allocate or release shared memory, so some ops are not supported
>> +as follows:
>> +
>> +- ``dev_configure``
>> +- ``dev_start``
>> +- ``dev_stop``
>> +- ``rx_queue_setup``
>> +- ``tx_queue_setup``
>> +- ``rx_queue_release``
>> +- ``tx_queue_release``
>> +
>> +So, any command from testpmd which calls those APIs will not be supported in
>> +secondary process, like:
>> +
>> +.. code-block:: console
>> +
>> + port config all rxq|txq|rxd|txd <value>
>> + port config <port_id> rx_offload xxx on/off
>> + port config <port_id> tx_offload xxx on/off
>> +
>> +etc.
>
> I did the formatting cleanup, but I still think that testpmd
> guide should not dive into such level of details. It should
> rather highlight multi-process behaviour specifics.
>
> Shouldn't testpmd store state in shared memory to avoid
> problems when primary is stopped while secondary is running
This could be taken into consideration in future.
>
> Some testpmd features rely on reconfigure (i.e. simply change
> configuration and set flag that reconfigure is required), but
> configure does nothing and will simply ignore new settings.
> So, it could look very-very confusing from user point of view.
>
> I'm not sure that it is acceptable to apply the patch in such
> state and open huge number of bugs in testpmd behaviour when
> multi-process is used.
>
> I'd even consider to exclude unsupported commands from help
> etc. However, such level of care about user could be excessive
> for test tool.
This has been done in doc.
>
> IMHO, it should be no requirement to repeat the primary
> process command-line configuration in the second process
> command line (see --rxq=4 --txq=4 above). The information
> should be obtained from shared state. In theory primary
> process could even change some settings in interactive
We think keeping the command line in consistent between primary
and secondary is easy to understand for users. While shared memory for
keeping in order or communicating could be performed,but this could
be done in future patch.
> mode. I think testpmd should guarantee consistent behaviour
> even in such conditions. I.e. do not allow to stop ports
> used by forwarding running in secondary processes.
> Run-time queues setup and deferred start should be very
> carefully handled as well.
``dev_stop`` is not allowed in secondary, which has described in doc.
>> +
>> +Stats is supported, stats will not change when one quits and starts, as they
>> +share the same buffer to store the stats. Flow rules are maintained in process
>> +level: primary and secondary has its own flow list (but one flow list in HW).
>> +The two can see all the queues, so setting the flow rules for the other is OK.
>> +But in the testpmd primary process receiving or transmitting packets from the
>> +queue allocated for secondary process is not permitted, and same for secondary
>> +process.
>> +
>> +Flow API and RSS are supported.
>>Thanks for your comment,
This patch supports basic function for multi-process support in testpmd.
I think other patches in future could enhance or optimize it, thanks.
>
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
2021-07-08 12:20 ` Min Hu (Connor)
@ 2021-07-08 12:30 ` Andrew Rybchenko
2021-07-08 12:51 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Andrew Rybchenko @ 2021-07-08 12:30 UTC (permalink / raw)
To: Min Hu (Connor), Xiaoyun Li, Anatoly Burakov
Cc: dev, Lijun Ou, Ajit Khaparde, Ferruh Yigit
On 7/8/21 3:20 PM, Min Hu (Connor) wrote:
> Hi, Andrew ,
>
> 在 2021/7/2 20:47, Andrew Rybchenko 写道:
>> On 7/2/21 3:09 PM, Andrew Rybchenko wrote:
>>> From: "Min Hu (Connor)" <humin29@huawei.com>
>>>
>>> For example the following commands run two testpmd processes:
>>>
>>> * the primary process:
>>>
>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>
>>> * the secondary process:
>>>
>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>
>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>
>> [snip]
>>
>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>>> index 1cdd3cdd1..a5da0c272 100644
>>> --- a/app/test-pmd/testpmd.c
>>> +++ b/app/test-pmd/testpmd.c
>>> @@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode =
>>> ETH_MQ_RX_VMDQ_DCB_RSS;
>>> */
>>> uint32_t eth_link_speed;
>>> +/*
>>> + * ID of the current process in multi-process, used to
>>> + * configure the queues to be polled.
>>> + */
>>> +int proc_id;
>>> +
>>> +/*
>>> + * Number of processes in multi-process, used to
>>> + * configure the queues to be polled.
>>> + */
>>> +unsigned int num_procs = 1;
>>> +
>>> +static int
>>> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t
>>> nb_tx_q,
>>> + const struct rte_eth_conf *dev_conf)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
>>> + dev_conf);
>>> + return 0;
>>> +}
>>> +
>>> +static int
>>> +eth_dev_start_mp(uint16_t port_id)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_start(port_id);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int
>>> +eth_dev_stop_mp(uint16_t port_id)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_stop(port_id);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static void
>>> +mempool_free_mp(struct rte_mempool *mp)
>>> +{
>>> + if (is_proc_primary())
>>> + rte_mempool_free(mp);
>>> +}
>>> +
>>> +static int
>>> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_set_mtu(port_id, mtu);
>>> +
>>> + return 0;
>>> +}
>>> +
>>
>> I think above functions should be removed and corresponding
>> checks should be done in caller directly since above functions
>> are used in single place only and just hide what actually
>> happens in the case of secondary process. It is very
>> misleading.
>>
> This was done as Ferruh suggested in V9， and this could reduce
> the complexity for testpmd when added by the multi-process support.
>> [snip]
>>
>>> @@ -2495,21 +2565,24 @@ start_port(portid_t pid)
>>> return -1;
>>> }
>>> /* configure port */
>>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>>> - nb_txq + nb_hairpinq,
>>> - &(port->dev_conf));
>>> + diag = eth_dev_configure_mp(pi,
>>> + nb_rxq + nb_hairpinq,
>>> + nb_txq + nb_hairpinq,
>>> + &(port->dev_conf));
>>> if (diag != 0) {
>>> - if (rte_atomic16_cmpset(&(port->port_status),
>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back "
>>> - "to stopped\n", pi);
>>> + if (rte_atomic16_cmpset(
>>> + &(port->port_status),
>>> + RTE_PORT_HANDLING,
>>> + RTE_PORT_STOPPED) == 0)
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided since
>> it just makes the review harder.This will be fixed in v16.
>>
>>> printf("Fail to configure port %d\n", pi);
>>> /* try to reconfigure port next time */
>>> port->need_reconfig = 1;
>>> return -1;
>>> }
>>> }
>>> - if (port->need_reconfig_queues > 0) {
>>> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
>>> port->need_reconfig_queues = 0;
>>> /* setup tx queues */
>>> for (qi = 0; qi < nb_txq; qi++) {
>>> @@ -2532,8 +2605,8 @@ start_port(portid_t pid)
>>> if (rte_atomic16_cmpset(&(port->port_status),
>>> RTE_PORT_HANDLING,
>>> RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back "
>>> - "to stopped\n", pi);
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided.
> This will be fixed in v16.
>>
>>> printf("Fail to configure port %d tx queues\n",
>>> pi);
>>> /* try to reconfigure queues next time */
>>> @@ -2610,16 +2683,16 @@ start_port(portid_t pid)
>>> cnt_pi++;
>>> /* start port */
>>> - diag = rte_eth_dev_start(pi);
>>> + diag = eth_dev_start_mp(pi);
>>> if (diag < 0) {
>>> printf("Fail to start port %d: %s\n", pi,
>>> rte_strerror(-diag));
>>> /* Fail to setup rx queue, return */
>>> if (rte_atomic16_cmpset(&(port->port_status),
>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back to "
>>> - "stopped\n", pi);
>>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided.
> This will be fixed in v16.
>>
>> [snip]
>>
>>> diff --git a/doc/guides/testpmd_app_ug/run_app.rst
>>> b/doc/guides/testpmd_app_ug/run_app.rst
>>> index eb4831835..348e5fcac 100644
>>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>>> @@ -545,3 +545,85 @@ The command line options are:
>>> bit 0 - two hairpin ports loop
>>> The default value is 0. Hairpin will use single port mode and
>>> implicit Tx flow mode.
>>> +
>>> +
>>> +Testpmd Multi-Process Command-line Options
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +The following are the command-line options for testpmd multi-process
>>> support:
>>> +
>>> +* primary process:
>>> +
>>> +.. code-block:: console
>>> +
>>> + sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4
>>> --txq=4 \
>>> + --num-procs=2 --proc-id=0
>>> +
>>> +* secondary process:
>>> +
>>> +.. code-block:: console
>>> +
>>> + sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4
>>> --txq=4 \
>>> + --num-procs=2 --proc-id=1
>>> +
>>> +The command line options are:
>>> +
>>> +* ``--num-procs=N``
>>> +
>>> + The number of processes which will be used.
>>> +
>>> +* ``--proc-id=ID``
>>> +
>>> + The ID of the current process (ID < num-procs). ID should be
>>> different in
>>> + primary process and secondary process, which starts from '0'.
>>> +
>>> +Calculation rule for queue:
>>> +All queues are allocated to different processes based on
>>> ``proc_num`` and
>>> +``proc_id``.
>>> +Calculation rule for the testpmd to allocate queues to each process:
>>> +
>>> +* start(queue start id) = proc_id * nb_q / num_procs
>>> +
>>> +* end(queue end id) = start + nb_q / num_procs
>>> +
>>> +For example, if testpmd is configured to have 4 Tx and Rx queues,
>>> +queues 0 and 1 will be used by the primary process and
>>> +queues 2 and 3 will be used by the secondary process.
>>> +
>>> +The number of queues should be a multiple of the number of
>>> processes. If not,
>>> +redundant queues will exist after queues are allocated to processes.
>>> If RSS
>>> +is enabled, packet loss occurs when traffic is sent to all processes
>>> at the same
>>> +time. Some traffic goes to redundant queues and cannot be forwarded.
>>> +
>>> +All the dev ops is supported in primary process. While secondary
>>> process is
>>> +not permitted to allocate or release shared memory, so some ops are
>>> not supported
>>> +as follows:
>>> +
>>> +- ``dev_configure``
>>> +- ``dev_start``
>>> +- ``dev_stop``
>>> +- ``rx_queue_setup``
>>> +- ``tx_queue_setup``
>>> +- ``rx_queue_release``
>>> +- ``tx_queue_release``
>>> +
>>> +So, any command from testpmd which calls those APIs will not be
>>> supported in
>>> +secondary process, like:
>>> +
>>> +.. code-block:: console
>>> +
>>> + port config all rxq|txq|rxd|txd <value>
>>> + port config <port_id> rx_offload xxx on/off
>>> + port config <port_id> tx_offload xxx on/off
>>> +
>>> +etc.
>>
>> I did the formatting cleanup, but I still think that testpmd
>> guide should not dive into such level of details. It should
>> rather highlight multi-process behaviour specifics.
>>
>> Shouldn't testpmd store state in shared memory to avoid
>> problems when primary is stopped while secondary is running
> This could be taken into consideration in future.
>
>>
>> Some testpmd features rely on reconfigure (i.e. simply change
>> configuration and set flag that reconfigure is required), but
>> configure does nothing and will simply ignore new settings.
>> So, it could look very-very confusing from user point of view.
>>
>> I'm not sure that it is acceptable to apply the patch in such
>> state and open huge number of bugs in testpmd behaviour when
>> multi-process is used.
>>
>> I'd even consider to exclude unsupported commands from help
>> etc. However, such level of care about user could be excessive
>> for test tool.
> This has been done in doc.
>>
>> IMHO, it should be no requirement to repeat the primary
>> process command-line configuration in the second process
>> command line (see --rxq=4 --txq=4 above). The information
>> should be obtained from shared state. In theory primary
>> process could even change some settings in interactive
> We think keeping the command line in consistent between primary
> and secondary is easy to understand for users. While shared memory for
> keeping in order or communicating could be performed,but this could
> be done in future patch.
>
>> mode. I think testpmd should guarantee consistent behaviour
>> even in such conditions. I.e. do not allow to stop ports
>> used by forwarding running in secondary processes.
>> Run-time queues setup and deferred start should be very
>> carefully handled as well.
> ``dev_stop`` is not allowed in secondary, which has described in doc.
I'm talking about dev_stop in primary while secondary is
running.
>>> +
>>> +Stats is supported, stats will not change when one quits and starts,
>>> as they
>>> +share the same buffer to store the stats. Flow rules are maintained
>>> in process
>>> +level: primary and secondary has its own flow list (but one flow
>>> list in HW).
>>> +The two can see all the queues, so setting the flow rules for the
>>> other is OK.
>>> +But in the testpmd primary process receiving or transmitting packets
>>> from the
>>> +queue allocated for secondary process is not permitted, and same for
>>> secondary
>>> +process.
>>> +
>>> +Flow API and RSS are supported.
>>> Thanks for your comment,
> This patch supports basic function for multi-process support in testpmd.
> I think other patches in future could enhance or optimize it, thanks.
IMHO, as I state above, current state is insufficient to
consider is a start point to be applied.
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
2021-07-08 12:30 ` Andrew Rybchenko
@ 2021-07-08 12:51 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-07-08 12:51 UTC (permalink / raw)
To: Andrew Rybchenko, Xiaoyun Li, Anatoly Burakov
Cc: dev, Lijun Ou, Ajit Khaparde, Ferruh Yigit
Hi,Andrew ,
在 2021/7/8 20:30, Andrew Rybchenko 写道:
> On 7/8/21 3:20 PM, Min Hu (Connor) wrote:
>> Hi, Andrew ,
>>
>> 在 2021/7/2 20:47, Andrew Rybchenko 写道:
>>> On 7/2/21 3:09 PM, Andrew Rybchenko wrote:
>>>> From: "Min Hu (Connor)" <humin29@huawei.com>
>>>>
>>>> For example the following commands run two testpmd processes:
>>>>
>>>> * the primary process:
>>>>
>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>
>>>> * the secondary process:
>>>>
>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>
>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>
>>> [snip]
>>>
>>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>>>> index 1cdd3cdd1..a5da0c272 100644
>>>> --- a/app/test-pmd/testpmd.c
>>>> +++ b/app/test-pmd/testpmd.c
>>>> @@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode =
>>>> ETH_MQ_RX_VMDQ_DCB_RSS;
>>>> */
>>>> uint32_t eth_link_speed;
>>>> +/*
>>>> + * ID of the current process in multi-process, used to
>>>> + * configure the queues to be polled.
>>>> + */
>>>> +int proc_id;
>>>> +
>>>> +/*
>>>> + * Number of processes in multi-process, used to
>>>> + * configure the queues to be polled.
>>>> + */
>>>> +unsigned int num_procs = 1;
>>>> +
>>>> +static int
>>>> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t
>>>> nb_tx_q,
>>>> + const struct rte_eth_conf *dev_conf)
>>>> +{
>>>> + if (is_proc_primary())
>>>> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
>>>> + dev_conf);
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int
>>>> +eth_dev_start_mp(uint16_t port_id)
>>>> +{
>>>> + if (is_proc_primary())
>>>> + return rte_eth_dev_start(port_id);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int
>>>> +eth_dev_stop_mp(uint16_t port_id)
>>>> +{
>>>> + if (is_proc_primary())
>>>> + return rte_eth_dev_stop(port_id);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void
>>>> +mempool_free_mp(struct rte_mempool *mp)
>>>> +{
>>>> + if (is_proc_primary())
>>>> + rte_mempool_free(mp);
>>>> +}
>>>> +
>>>> +static int
>>>> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
>>>> +{
>>>> + if (is_proc_primary())
>>>> + return rte_eth_dev_set_mtu(port_id, mtu);
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>
>>> I think above functions should be removed and corresponding
>>> checks should be done in caller directly since above functions
>>> are used in single place only and just hide what actually
>>> happens in the case of secondary process. It is very
>>> misleading.
>>>
>> This was done as Ferruh suggested in V9， and this could reduce
>> the complexity for testpmd when added by the multi-process support.
>>> [snip]
>>>
>>>> @@ -2495,21 +2565,24 @@ start_port(portid_t pid)
>>>> return -1;
>>>> }
>>>> /* configure port */
>>>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>>>> - nb_txq + nb_hairpinq,
>>>> - &(port->dev_conf));
>>>> + diag = eth_dev_configure_mp(pi,
>>>> + nb_rxq + nb_hairpinq,
>>>> + nb_txq + nb_hairpinq,
>>>> + &(port->dev_conf));
>>>> if (diag != 0) {
>>>> - if (rte_atomic16_cmpset(&(port->port_status),
>>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>>> - printf("Port %d can not be set back "
>>>> - "to stopped\n", pi);
>>>> + if (rte_atomic16_cmpset(
>>>> + &(port->port_status),
>>>> + RTE_PORT_HANDLING,
>>>> + RTE_PORT_STOPPED) == 0)
>>>> + printf("Port %d cannot be set back to stopped\n",
>>>> + pi);
>>>
>>> Unrelated changes in the patch should be avoided since
>>> it just makes the review harder.This will be fixed in v16.
>>>
>>>> printf("Fail to configure port %d\n", pi);
>>>> /* try to reconfigure port next time */
>>>> port->need_reconfig = 1;
>>>> return -1;
>>>> }
>>>> }
>>>> - if (port->need_reconfig_queues > 0) {
>>>> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
>>>> port->need_reconfig_queues = 0;
>>>> /* setup tx queues */
>>>> for (qi = 0; qi < nb_txq; qi++) {
>>>> @@ -2532,8 +2605,8 @@ start_port(portid_t pid)
>>>> if (rte_atomic16_cmpset(&(port->port_status),
>>>> RTE_PORT_HANDLING,
>>>> RTE_PORT_STOPPED) == 0)
>>>> - printf("Port %d can not be set back "
>>>> - "to stopped\n", pi);
>>>> + printf("Port %d cannot be set back to stopped\n",
>>>> + pi);
>>>
>>> Unrelated changes in the patch should be avoided.
>> This will be fixed in v16.
>>>
>>>> printf("Fail to configure port %d tx queues\n",
>>>> pi);
>>>> /* try to reconfigure queues next time */
>>>> @@ -2610,16 +2683,16 @@ start_port(portid_t pid)
>>>> cnt_pi++;
>>>> /* start port */
>>>> - diag = rte_eth_dev_start(pi);
>>>> + diag = eth_dev_start_mp(pi);
>>>> if (diag < 0) {
>>>> printf("Fail to start port %d: %s\n", pi,
>>>> rte_strerror(-diag));
>>>> /* Fail to setup rx queue, return */
>>>> if (rte_atomic16_cmpset(&(port->port_status),
>>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>>> - printf("Port %d can not be set back to "
>>>> - "stopped\n", pi);
>>>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>>> + printf("Port %d cannot be set back to stopped\n",
>>>> + pi);
>>>
>>> Unrelated changes in the patch should be avoided.
>> This will be fixed in v16.
>>>
>>> [snip]
>>>
>>>> diff --git a/doc/guides/testpmd_app_ug/run_app.rst
>>>> b/doc/guides/testpmd_app_ug/run_app.rst
>>>> index eb4831835..348e5fcac 100644
>>>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>>>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>>>> @@ -545,3 +545,85 @@ The command line options are:
>>>> bit 0 - two hairpin ports loop
>>>> The default value is 0. Hairpin will use single port mode and
>>>> implicit Tx flow mode.
>>>> +
>>>> +
>>>> +Testpmd Multi-Process Command-line Options
>>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>> +
>>>> +The following are the command-line options for testpmd multi-process
>>>> support:
>>>> +
>>>> +* primary process:
>>>> +
>>>> +.. code-block:: console
>>>> +
>>>> + sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4
>>>> --txq=4 \
>>>> + --num-procs=2 --proc-id=0
>>>> +
>>>> +* secondary process:
>>>> +
>>>> +.. code-block:: console
>>>> +
>>>> + sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4
>>>> --txq=4 \
>>>> + --num-procs=2 --proc-id=1
>>>> +
>>>> +The command line options are:
>>>> +
>>>> +* ``--num-procs=N``
>>>> +
>>>> + The number of processes which will be used.
>>>> +
>>>> +* ``--proc-id=ID``
>>>> +
>>>> + The ID of the current process (ID < num-procs). ID should be
>>>> different in
>>>> + primary process and secondary process, which starts from '0'.
>>>> +
>>>> +Calculation rule for queue:
>>>> +All queues are allocated to different processes based on
>>>> ``proc_num`` and
>>>> +``proc_id``.
>>>> +Calculation rule for the testpmd to allocate queues to each process:
>>>> +
>>>> +* start(queue start id) = proc_id * nb_q / num_procs
>>>> +
>>>> +* end(queue end id) = start + nb_q / num_procs
>>>> +
>>>> +For example, if testpmd is configured to have 4 Tx and Rx queues,
>>>> +queues 0 and 1 will be used by the primary process and
>>>> +queues 2 and 3 will be used by the secondary process.
>>>> +
>>>> +The number of queues should be a multiple of the number of
>>>> processes. If not,
>>>> +redundant queues will exist after queues are allocated to processes.
>>>> If RSS
>>>> +is enabled, packet loss occurs when traffic is sent to all processes
>>>> at the same
>>>> +time. Some traffic goes to redundant queues and cannot be forwarded.
>>>> +
>>>> +All the dev ops is supported in primary process. While secondary
>>>> process is
>>>> +not permitted to allocate or release shared memory, so some ops are
>>>> not supported
>>>> +as follows:
>>>> +
>>>> +- ``dev_configure``
>>>> +- ``dev_start``
>>>> +- ``dev_stop``
>>>> +- ``rx_queue_setup``
>>>> +- ``tx_queue_setup``
>>>> +- ``rx_queue_release``
>>>> +- ``tx_queue_release``
>>>> +
>>>> +So, any command from testpmd which calls those APIs will not be
>>>> supported in
>>>> +secondary process, like:
>>>> +
>>>> +.. code-block:: console
>>>> +
>>>> + port config all rxq|txq|rxd|txd <value>
>>>> + port config <port_id> rx_offload xxx on/off
>>>> + port config <port_id> tx_offload xxx on/off
>>>> +
>>>> +etc.
>>>
>>> I did the formatting cleanup, but I still think that testpmd
>>> guide should not dive into such level of details. It should
>>> rather highlight multi-process behaviour specifics.
>>>
>>> Shouldn't testpmd store state in shared memory to avoid
>>> problems when primary is stopped while secondary is running
>> This could be taken into consideration in future.
>>
>>>
>>> Some testpmd features rely on reconfigure (i.e. simply change
>>> configuration and set flag that reconfigure is required), but
>>> configure does nothing and will simply ignore new settings.
>>> So, it could look very-very confusing from user point of view.
>>>
>>> I'm not sure that it is acceptable to apply the patch in such
>>> state and open huge number of bugs in testpmd behaviour when
>>> multi-process is used.
>>>
>>> I'd even consider to exclude unsupported commands from help
>>> etc. However, such level of care about user could be excessive
>>> for test tool.
>> This has been done in doc.
>>>
>>> IMHO, it should be no requirement to repeat the primary
>>> process command-line configuration in the second process
>>> command line (see --rxq=4 --txq=4 above). The information
>>> should be obtained from shared state. In theory primary
>>> process could even change some settings in interactive
>> We think keeping the command line in consistent between primary
>> and secondary is easy to understand for users. While shared memory for
>> keeping in order or communicating could be performed,but this could
>> be done in future patch.
>>
>>> mode. I think testpmd should guarantee consistent behaviour
>>> even in such conditions. I.e. do not allow to stop ports
>>> used by forwarding running in secondary processes.
>>> Run-time queues setup and deferred start should be very
>>> carefully handled as well.
>> ``dev_stop`` is not allowed in secondary, which has described in doc.
>
> I'm talking about dev_stop in primary while secondary is
> running.
>
>>>> +
>>>> +Stats is supported, stats will not change when one quits and starts,
>>>> as they
>>>> +share the same buffer to store the stats. Flow rules are maintained
>>>> in process
>>>> +level: primary and secondary has its own flow list (but one flow
>>>> list in HW).
>>>> +The two can see all the queues, so setting the flow rules for the
>>>> other is OK.
>>>> +But in the testpmd primary process receiving or transmitting packets
>>>> from the
>>>> +queue allocated for secondary process is not permitted, and same for
>>>> secondary
>>>> +process.
>>>> +
>>>> +Flow API and RSS are supported.
>>>> Thanks for your comment,
>> This patch supports basic function for multi-process support in testpmd.
>> I think other patches in future could enhance or optimize it, thanks.
>
> IMHO, as I state above, current state is insufficient to
> consider is a start point to be applied.
OK, what is the critical(the most important) as a start point to be applied?
I will take action to fix it,as I will the patch could be applied before
V21.08.
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
` (2 preceding siblings ...)
2021-07-02 12:09 ` [dpdk-dev] [PATCH v15] " Andrew Rybchenko
@ 2021-07-10 3:50 ` Min Hu (Connor)
2021-07-24 11:45 ` Thomas Monjalon
2021-08-25 2:06 ` [dpdk-dev] [PATCH v17] " Min Hu (Connor)
4 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-07-10 3:50 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, andrew.rybchenko
This patch adds multi-process support for testpmd.
For example the following commands run two testpmd
processes:
* the primary process:
./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
* the secondary process:
./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
V16:
* revert unrelated changes.
* add some restrictions in doc.
v15:
* Fixed release notes.
* Cleanup documentation.
v14:
* Fixed comments by Andrew Rybchenko.
v13:
* Modified the doc syntax.
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 20 ++++++-
app/test-pmd/parameters.c | 9 +++
app/test-pmd/testpmd.c | 106 ++++++++++++++++++++++++++++-----
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_08.rst | 3 +
doc/guides/testpmd_app_ug/run_app.rst | 84 ++++++++++++++++++++++++++
7 files changed, 220 insertions(+), 17 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 8468018..5b9880d 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5459,6 +5459,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush Rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 04ae0fe..9e4d86b 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2976,6 +2976,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2993,7 +2995,21 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs != 0)
+ printf("Warning! queue numbers should be multiple of processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -3010,6 +3026,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 5e69d2a..925fca5 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -509,6 +509,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -628,6 +631,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1394,6 +1399,10 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (!strcmp(lgopts[opt_idx].name, PARAM_NUM_PROCS))
+ num_procs = atoi(optarg);
+ if (!strcmp(lgopts[opt_idx].name, PARAM_PROC_ID))
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1cdd3cd..731a6a6 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * ID of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -982,6 +1038,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
#endif
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ if (rte_mp == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ return rte_mp;
+ }
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
@@ -2008,6 +2072,11 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd Rx queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2495,12 +2564,15 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
printf("Port %d can not be set back "
"to stopped\n", pi);
printf("Fail to configure port %d\n", pi);
@@ -2509,7 +2581,7 @@ start_port(portid_t pid)
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2610,7 +2682,7 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
@@ -2744,7 +2816,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2813,8 +2885,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3082,7 +3156,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3413,7 +3487,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3603,6 +3677,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3771,10 +3849,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index d61a055..11b9451 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_08.rst b/doc/guides/rel_notes/release_21_08.rst
index 0804b00..669cb75 100644
--- a/doc/guides/rel_notes/release_21_08.rst
+++ b/doc/guides/rel_notes/release_21_08.rst
@@ -90,6 +90,9 @@ New Features
usecases. Configuration happens via standard rawdev enq/deq operations. See
the :doc:`../rawdevs/cnxk_bphy` rawdev guide for more details on this driver.
+* **Added support multi-process for testpmd.**
+ Added command-line options to specify total number of processes and
+ current process ID. Each process owns subset of Rx and Tx queues.
Removed Items
-------------
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index eb48318..f98e5b9 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -545,3 +545,87 @@ The command line options are:
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+* primary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+* secondary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=ID``
+
+ The ID of the current process (ID < num-procs). ID should be different in
+ primary process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and
+``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+* start(queue start id) = proc_id * nb_q / num_procs；
+
+* end(queue end id) = start + nb_q / num_procs；
+
+For example, if testpmd is configured to have 4 Tx and Rx queues,
+queues 0 and 1 will be used by the primary process and
+queues 2 and 3 will be used by the secondary process.
+
+The number of queues should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. If RSS
+is enabled, packet loss occurs when traffic is sent to all processes at the same
+time. Some traffic goes to redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is
+not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+
+- ``dev_configure``
+- ``dev_start``
+- ``dev_stop``
+- ``rx_queue_setup``
+- ``tx_queue_setup``
+- ``rx_queue_release``
+- ``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in
+secondary process, like:
+
+.. code-block:: console
+
+ port config all rxq|txq|rxd|txd <value>
+ port config <port_id> rx_offload xxx on/off
+ port config <port_id> tx_offload xxx on/off
+
+etc.
+
+When secondary is running, port in primary is not permitted to be stopped.
+Reconfigure operation is only valid in primary.
+
+Stats is supported, stats will not change when one quits and starts, as they
+share the same buffer to store the stats. Flow rules are maintained in process
+level: primary and secondary has its own flow list (but one flow list in HW).
+The two can see all the queues, so setting the flow rules for the other is OK.
+But in the testpmd primary process receiving or transmitting packets from the
+queue allocated for secondary process is not permitted, and same for secondary
+process.
+
+Flow API and RSS are supported.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-07-10 3:50 ` [dpdk-dev] [PATCH v16] " Min Hu (Connor)
@ 2021-07-24 11:45 ` Thomas Monjalon
2021-07-26 0:26 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Monjalon @ 2021-07-24 11:45 UTC (permalink / raw)
To: Min Hu (Connor)
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
10/07/2021 05:50, Min Hu (Connor):
> This patch adds multi-process support for testpmd.
> For example the following commands run two testpmd
> processes:
>
> * the primary process:
>
> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> * the secondary process:
>
> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> V16:
> * revert unrelated changes.
> * add some restrictions in doc.
I didn't see clear agreement to integrate this feature in DPDK 21.08.
BTW, the testpmd maintainer was not Cc'ed.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-07-24 11:45 ` Thomas Monjalon
@ 2021-07-26 0:26 ` Min Hu (Connor)
2021-07-26 6:30 ` Thomas Monjalon
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-07-26 0:26 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
在 2021/7/24 19:45, Thomas Monjalon 写道:
> 10/07/2021 05:50, Min Hu (Connor):
>> This patch adds multi-process support for testpmd.
>> For example the following commands run two testpmd
>> processes:
>>
>> * the primary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> * the secondary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> V16:
>> * revert unrelated changes.
>> * add some restrictions in doc.
>
> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>
> BTW, the testpmd maintainer was not Cc'ed.
>
Hi, Thomas,
This patch has two "acked" and one "reviewed-by".
And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
please check it out, thanks.
>
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-07-26 0:26 ` Min Hu (Connor)
@ 2021-07-26 6:30 ` Thomas Monjalon
2021-07-26 7:28 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Monjalon @ 2021-07-26 6:30 UTC (permalink / raw)
To: Min Hu (Connor)
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
26/07/2021 02:26, Min Hu (Connor):
> 在 2021/7/24 19:45, Thomas Monjalon 写道:
> > 10/07/2021 05:50, Min Hu (Connor):
> >> This patch adds multi-process support for testpmd.
> >> For example the following commands run two testpmd
> >> processes:
> >>
> >> * the primary process:
> >>
> >> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
> >> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
> >>
> >> * the secondary process:
> >>
> >> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
> >> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
> >>
> >> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> >> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> >> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
> >> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> >> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >> ---
> >> V16:
> >> * revert unrelated changes.
> >> * add some restrictions in doc.
> >
> > I didn't see clear agreement to integrate this feature in DPDK 21.08.
> >
> > BTW, the testpmd maintainer was not Cc'ed.
> >
> Hi, Thomas,
> This patch has two "acked" and one "reviewed-by".
> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
> please check it out, thanks.
Andrew's comment from v15:
"
IMHO, as I state above, current state is insufficient to
consider is a start point to be applied.
"
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-07-26 6:30 ` Thomas Monjalon
@ 2021-07-26 7:28 ` Min Hu (Connor)
2021-08-02 1:51 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-07-26 7:28 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
在 2021/7/26 14:30, Thomas Monjalon 写道:
> 26/07/2021 02:26, Min Hu (Connor):
>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
>>> 10/07/2021 05:50, Min Hu (Connor):
>>>> This patch adds multi-process support for testpmd.
>>>> For example the following commands run two testpmd
>>>> processes:
>>>>
>>>> * the primary process:
>>>>
>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>
>>>> * the secondary process:
>>>>
>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>
>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> ---
>>>> V16:
>>>> * revert unrelated changes.
>>>> * add some restrictions in doc.
>>>
>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>>>
>>> BTW, the testpmd maintainer was not Cc'ed.
>>>
>> Hi, Thomas,
>> This patch has two "acked" and one "reviewed-by".
>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
>> please check it out, thanks.
>
> Andrew's comment from v15:
> "
> IMHO, as I state above, current state is insufficient to
> consider is a start point to be applied.
> "
Hi, Andrew, Thomos,
What is the least thing should be done for this patch to
meet the state sufficient to be applied?
>
>
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-07-26 7:28 ` Min Hu (Connor)
@ 2021-08-02 1:51 ` Min Hu (Connor)
2021-08-02 8:03 ` Thomas Monjalon
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-08-02 1:51 UTC (permalink / raw)
To: Thomas Monjalon
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
在 2021/7/26 15:28, Min Hu (Connor) 写道:
>
>
> 在 2021/7/26 14:30, Thomas Monjalon 写道:
>> 26/07/2021 02:26, Min Hu (Connor):
>>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
>>>> 10/07/2021 05:50, Min Hu (Connor):
>>>>> This patch adds multi-process support for testpmd.
>>>>> For example the following commands run two testpmd
>>>>> processes:
>>>>>
>>>>> * the primary process:
>>>>>
>>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>
>>>>> * the secondary process:
>>>>>
>>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>
>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>> ---
>>>>> V16:
>>>>> * revert unrelated changes.
>>>>> * add some restrictions in doc.
>>>>
>>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>>>>
>>>> BTW, the testpmd maintainer was not Cc'ed.
>>>>
>>> Hi, Thomas,
>>> This patch has two "acked" and one "reviewed-by".
>>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
>>> please check it out, thanks.
>>
>> Andrew's comment from v15:
>> "
>> IMHO, as I state above, current state is insufficient to
>> consider is a start point to be applied.
>> "
> Hi, Andrew, Thomos,
> What is the least thing should be done for this patch to
> meet the state sufficient to be applied?
Hi, all,
How about this patch, could it be applied?
>
>>
>>
>> .
>>
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-08-02 1:51 ` Min Hu (Connor)
@ 2021-08-02 8:03 ` Thomas Monjalon
2021-08-16 18:12 ` Singh, Aman Deep
0 siblings, 1 reply; 80+ messages in thread
From: Thomas Monjalon @ 2021-08-02 8:03 UTC (permalink / raw)
To: Min Hu (Connor)
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
02/08/2021 03:51, Min Hu (Connor):
> 在 2021/7/26 15:28, Min Hu (Connor) 写道:
> > 在 2021/7/26 14:30, Thomas Monjalon 写道:
> >> 26/07/2021 02:26, Min Hu (Connor):
> >>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
> >>>> 10/07/2021 05:50, Min Hu (Connor):
> >>>>> This patch adds multi-process support for testpmd.
> >>>>> For example the following commands run two testpmd
> >>>>> processes:
> >>>>>
> >>>>> * the primary process:
> >>>>>
> >>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
> >>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
> >>>>>
> >>>>> * the secondary process:
> >>>>>
> >>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
> >>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
> >>>>>
> >>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> >>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> >>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
> >>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> >>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> >>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >>>>> ---
> >>>>> V16:
> >>>>> * revert unrelated changes.
> >>>>> * add some restrictions in doc.
> >>>>
> >>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
> >>>>
> >>>> BTW, the testpmd maintainer was not Cc'ed.
> >>>>
> >>> Hi, Thomas,
> >>> This patch has two "acked" and one "reviewed-by".
> >>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
> >>> please check it out, thanks.
> >>
> >> Andrew's comment from v15:
> >> "
> >> IMHO, as I state above, current state is insufficient to
> >> consider is a start point to be applied.
> >> "
> > Hi, Andrew, Thomos,
> > What is the least thing should be done for this patch to
> > meet the state sufficient to be applied?
> Hi, all,
> How about this patch, could it be applied?
I was not confident enough to apply it in 21.08.
When others will be back, we'll discuss it again for 21.11.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-08-02 8:03 ` Thomas Monjalon
@ 2021-08-16 18:12 ` Singh, Aman Deep
2021-08-24 12:18 ` Ferruh Yigit
0 siblings, 1 reply; 80+ messages in thread
From: Singh, Aman Deep @ 2021-08-16 18:12 UTC (permalink / raw)
To: Thomas Monjalon, Min Hu (Connor)
Cc: dev, ferruh.yigit, andrew.rybchenko, david.marchand, Xiaoyun Li,
Ajit Khaparde, viacheslavo
On 8/2/2021 1:33 PM, Thomas Monjalon wrote:
> 02/08/2021 03:51, Min Hu (Connor):
>> 在 2021/7/26 15:28, Min Hu (Connor) 写道:
>>> 在 2021/7/26 14:30, Thomas Monjalon 写道:
>>>> 26/07/2021 02:26, Min Hu (Connor):
>>>>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
>>>>>> 10/07/2021 05:50, Min Hu (Connor):
>>>>>>> This patch adds multi-process support for testpmd.
>>>>>>> For example the following commands run two testpmd
>>>>>>> processes:
>>>>>>>
>>>>>>> * the primary process:
>>>>>>>
>>>>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>>>
>>>>>>> * the secondary process:
>>>>>>>
>>>>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>>>
>>>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>>>> ---
>>>>>>> V16:
>>>>>>> * revert unrelated changes.
>>>>>>> * add some restrictions in doc.
>>>>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>>>>>>
>>>>>> BTW, the testpmd maintainer was not Cc'ed.
>>>>>>
>>>>> Hi, Thomas,
>>>>> This patch has two "acked" and one "reviewed-by".
>>>>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
>>>>> please check it out, thanks.
>>>> Andrew's comment from v15:
>>>> "
>>>> IMHO, as I state above, current state is insufficient to
>>>> consider is a start point to be applied.
>>>> "
>>> Hi, Andrew, Thomos,
>>> What is the least thing should be done for this patch to
>>> meet the state sufficient to be applied?
>> Hi, all,
>> How about this patch, could it be applied?
> I was not confident enough to apply it in 21.08.
> When others will be back, we'll discuss it again for 21.11.
>
For 21.11 release.
Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-08-16 18:12 ` Singh, Aman Deep
@ 2021-08-24 12:18 ` Ferruh Yigit
2021-08-24 13:27 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-08-24 12:18 UTC (permalink / raw)
To: Singh, Aman Deep, Thomas Monjalon, Min Hu (Connor)
Cc: dev, andrew.rybchenko, david.marchand, Xiaoyun Li, Ajit Khaparde,
viacheslavo
On 8/16/2021 7:12 PM, Singh, Aman Deep wrote:
>
> On 8/2/2021 1:33 PM, Thomas Monjalon wrote:
>> 02/08/2021 03:51, Min Hu (Connor):
>>> 在 2021/7/26 15:28, Min Hu (Connor) 写道:
>>>> 在 2021/7/26 14:30, Thomas Monjalon 写道:
>>>>> 26/07/2021 02:26, Min Hu (Connor):
>>>>>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
>>>>>>> 10/07/2021 05:50, Min Hu (Connor):
>>>>>>>> This patch adds multi-process support for testpmd.
>>>>>>>> For example the following commands run two testpmd
>>>>>>>> processes:
>>>>>>>>
>>>>>>>> * the primary process:
>>>>>>>>
>>>>>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>>>>
>>>>>>>> * the secondary process:
>>>>>>>>
>>>>>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>>>>
>>>>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>>>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>>>>> ---
>>>>>>>> V16:
>>>>>>>> * revert unrelated changes.
>>>>>>>> * add some restrictions in doc.
>>>>>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>>>>>>>
>>>>>>> BTW, the testpmd maintainer was not Cc'ed.
>>>>>>>
>>>>>> Hi, Thomas,
>>>>>> This patch has two "acked" and one "reviewed-by".
>>>>>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
>>>>>> please check it out, thanks.
>>>>> Andrew's comment from v15:
>>>>> "
>>>>> IMHO, as I state above, current state is insufficient to
>>>>> consider is a start point to be applied.
>>>>> "
>>>> Hi, Andrew, Thomos,
>>>> What is the least thing should be done for this patch to
>>>> meet the state sufficient to be applied?
>>> Hi, all,
>>> How about this patch, could it be applied?
>> I was not confident enough to apply it in 21.08.
>> When others will be back, we'll discuss it again for 21.11.
>>
> For 21.11 release.
>
> Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
>
Hi Connor,
Can you please rebase it to latest dpdk (v21.11-rc0) so we can proceed with it?
I am for getting the patch as early as possible in the release cycle, specially
before it is getting busy for -rc1.
Thanks,
ferruh
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v16] app/testpmd: support multi-process
2021-08-24 12:18 ` Ferruh Yigit
@ 2021-08-24 13:27 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-08-24 13:27 UTC (permalink / raw)
To: Ferruh Yigit, Singh, Aman Deep, Thomas Monjalon
Cc: dev, andrew.rybchenko, david.marchand, Xiaoyun Li, Ajit Khaparde,
viacheslavo
Thanks Ferruh, I will handle it as quickly as possible.
在 2021/8/24 20:18, Ferruh Yigit 写道:
> On 8/16/2021 7:12 PM, Singh, Aman Deep wrote:
>>
>> On 8/2/2021 1:33 PM, Thomas Monjalon wrote:
>>> 02/08/2021 03:51, Min Hu (Connor):
>>>> 在 2021/7/26 15:28, Min Hu (Connor) 写道:
>>>>> 在 2021/7/26 14:30, Thomas Monjalon 写道:
>>>>>> 26/07/2021 02:26, Min Hu (Connor):
>>>>>>> 在 2021/7/24 19:45, Thomas Monjalon 写道:
>>>>>>>> 10/07/2021 05:50, Min Hu (Connor):
>>>>>>>>> This patch adds multi-process support for testpmd.
>>>>>>>>> For example the following commands run two testpmd
>>>>>>>>> processes:
>>>>>>>>>
>>>>>>>>> * the primary process:
>>>>>>>>>
>>>>>>>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>>>>>
>>>>>>>>> * the secondary process:
>>>>>>>>>
>>>>>>>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>>>>>
>>>>>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>>>>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>>>>>>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>>>>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>>>>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>>>>>> ---
>>>>>>>>> V16:
>>>>>>>>> * revert unrelated changes.
>>>>>>>>> * add some restrictions in doc.
>>>>>>>> I didn't see clear agreement to integrate this feature in DPDK 21.08.
>>>>>>>>
>>>>>>>> BTW, the testpmd maintainer was not Cc'ed.
>>>>>>>>
>>>>>>> Hi, Thomas,
>>>>>>> This patch has two "acked" and one "reviewed-by".
>>>>>>> And Xiaoyun Li is the maintainer, I has already Cc'ed to him/her,
>>>>>>> please check it out, thanks.
>>>>>> Andrew's comment from v15:
>>>>>> "
>>>>>> IMHO, as I state above, current state is insufficient to
>>>>>> consider is a start point to be applied.
>>>>>> "
>>>>> Hi, Andrew, Thomos,
>>>>> What is the least thing should be done for this patch to
>>>>> meet the state sufficient to be applied?
>>>> Hi, all,
>>>> How about this patch, could it be applied?
>>> I was not confident enough to apply it in 21.08.
>>> When others will be back, we'll discuss it again for 21.11.
>>>
>> For 21.11 release.
>>
>> Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
>>
>
> Hi Connor,
>
> Can you please rebase it to latest dpdk (v21.11-rc0) so we can proceed with it?
>
> I am for getting the patch as early as possible in the release cycle, specially
> before it is getting busy for -rc1.
>
> Thanks,
> ferruh
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v17] app/testpmd: support multi-process
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
` (3 preceding siblings ...)
2021-07-10 3:50 ` [dpdk-dev] [PATCH v16] " Min Hu (Connor)
@ 2021-08-25 2:06 ` Min Hu (Connor)
2021-09-07 13:23 ` Ferruh Yigit
4 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-08-25 2:06 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, aman.deep.singh, thomas
This patch adds multi-process support for testpmd.
For example the following commands run two testpmd
processes:
* the primary process:
./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
* the secondary process:
./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
---
v17:
rebase 21.11
V16:
* revert unrelated changes.
* add some restrictions in doc.
v15:
* Fixed release notes.
* Cleanup documentation.
v14:
* Fixed comments by Andrew Rybchenko.
v13:
* Modified the doc syntax.
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 +++
app/test-pmd/config.c | 20 ++++++-
app/test-pmd/parameters.c | 9 ++++
app/test-pmd/testpmd.c | 95 +++++++++++++++++++++++++++++-----
app/test-pmd/testpmd.h | 9 ++++
doc/guides/rel_notes/release_21_11.rst | 4 ++
doc/guides/testpmd_app_ug/run_app.rst | 84 ++++++++++++++++++++++++++++++
7 files changed, 214 insertions(+), 13 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 82253bc..395122b 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5491,6 +5491,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush Rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 31d8ba1..bd404db 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2998,6 +2998,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -3015,7 +3017,21 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs != 0)
+ printf("Warning! queue numbers should be multiple of processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -3032,6 +3048,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index 7c13210..26192f1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -512,6 +512,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -631,6 +634,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1397,6 +1402,10 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (!strcmp(lgopts[opt_idx].name, PARAM_NUM_PROCS))
+ num_procs = atoi(optarg);
+ if (!strcmp(lgopts[opt_idx].name, PARAM_PROC_ID))
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 6cbe9ba..66bce4c 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * ID of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -995,6 +1051,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
#endif
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ if (rte_mp == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ return rte_mp;
+ }
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
@@ -1994,6 +2058,11 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd Rx queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2483,7 +2552,7 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ diag = eth_dev_configure_mp(pi, nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
if (diag != 0) {
@@ -2499,7 +2568,7 @@ start_port(portid_t pid)
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2602,7 +2671,7 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
fprintf(stderr, "Fail to start port %d: %s\n",
pi, rte_strerror(-diag));
@@ -2745,7 +2814,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2819,8 +2888,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3105,7 +3176,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3446,7 +3517,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
fprintf(stderr,
@@ -3642,6 +3713,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3811,10 +3886,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 16a3598..04773c7 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index d707a55..2a9d3b9 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -51,6 +51,10 @@ New Features
* Other libs
* Apps, Examples, Tools (if significant)
+ * **Added support multi-process for testpmd.**
+ Added command-line options to specify total number of processes and
+ current process ID. Each process owns subset of Rx and Tx queues.
+
This section is a comment. Do not overwrite or remove it.
Also, make sure to start the actual text at the margin.
=======================================================
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 6061674..ebbe8d9 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -545,3 +545,87 @@ The command line options are:
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+* primary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+* secondary process:
+
+.. code-block:: console
+
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=ID``
+
+ The ID of the current process (ID < num-procs). ID should be different in
+ primary process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and
+``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+* start(queue start id) = proc_id * nb_q / num_procs；
+
+* end(queue end id) = start + nb_q / num_procs；
+
+For example, if testpmd is configured to have 4 Tx and Rx queues,
+queues 0 and 1 will be used by the primary process and
+queues 2 and 3 will be used by the secondary process.
+
+The number of queues should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. If RSS
+is enabled, packet loss occurs when traffic is sent to all processes at the same
+time. Some traffic goes to redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is
+not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+
+- ``dev_configure``
+- ``dev_start``
+- ``dev_stop``
+- ``rx_queue_setup``
+- ``tx_queue_setup``
+- ``rx_queue_release``
+- ``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in
+secondary process, like:
+
+.. code-block:: console
+
+ port config all rxq|txq|rxd|txd <value>
+ port config <port_id> rx_offload xxx on/off
+ port config <port_id> tx_offload xxx on/off
+
+etc.
+
+When secondary is running, port in primary is not permitted to be stopped.
+Reconfigure operation is only valid in primary.
+
+Stats is supported, stats will not change when one quits and starts, as they
+share the same buffer to store the stats. Flow rules are maintained in process
+level: primary and secondary has its own flow list (but one flow list in HW).
+The two can see all the queues, so setting the flow rules for the other is OK.
+But in the testpmd primary process receiving or transmitting packets from the
+queue allocated for secondary process is not permitted, and same for secondary
+process.
+
+Flow API and RSS are supported.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v17] app/testpmd: support multi-process
2021-08-25 2:06 ` [dpdk-dev] [PATCH v17] " Min Hu (Connor)
@ 2021-09-07 13:23 ` Ferruh Yigit
2021-09-08 0:48 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-09-07 13:23 UTC (permalink / raw)
To: Min Hu (Connor), Xiaoyun Li
Cc: aman.deep.singh, thomas, Andrew Rybchenko, Ajit Khaparde, dev
On 8/25/2021 3:06 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> For example the following commands run two testpmd
> processes:
>
> * the primary process:
>
> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> * the secondary process:
>
> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
Applied to dpdk-next-net/main, thanks.
Thanks Connor for the work,
I won't be surprised if we need a few more tweak/fix for the testpmd secondary
process support but right now there is no blocker issue, so proceeding with the
patch, please continue your support for further required changes.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v17] app/testpmd: support multi-process
2021-09-07 13:23 ` Ferruh Yigit
@ 2021-09-08 0:48 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-09-08 0:48 UTC (permalink / raw)
To: Ferruh Yigit, Xiaoyun Li
Cc: aman.deep.singh, thomas, Andrew Rybchenko, Ajit Khaparde, dev
在 2021/9/7 21:23, Ferruh Yigit 写道:
> On 8/25/2021 3:06 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> For example the following commands run two testpmd
>> processes:
>>
>> * the primary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> * the secondary process:
>>
>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Acked-by: Aman Deep Singh <aman.deep.singh@intel.com>
>
> Applied to dpdk-next-net/main, thanks.
>
>
> Thanks Connor for the work,
> I won't be surprised if we need a few more tweak/fix for the testpmd secondary
> process support but right now there is no blocker issue, so proceeding with the
> patch, please continue your support for further required changes.
Yes, got it, thanks Ferruh.
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v3] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
2021-03-05 4:05 ` Ajit Khaparde
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
@ 2021-03-11 9:07 ` Min Hu (Connor)
2021-03-20 0:58 ` Min Hu (Connor)
2021-03-22 7:07 ` [dpdk-dev] [PATCH v4] " Min Hu (Connor)
` (9 subsequent siblings)
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-11 9:07 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
app/test-pmd/cmdline.c | 12 ++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
6 files changed, 196 insertions(+), 50 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 576d5ac..9edf84a 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c8acd5d..1a5bf25 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -605,6 +607,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1366,6 +1370,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1a57324..bbd45d9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,8 @@
#include "testpmd.h"
+int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -982,16 +987,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
/* wrapper to rte_mempool_create() */
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size, socket_id);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, socket_id);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_ANON:
{
- rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf,
- mb_size, (unsigned int) mb_mempool_cache,
- sizeof(struct rte_pktmbuf_pool_private),
- socket_id, mempool_flags);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_mempool_create_empty(pool_name,
+ nb_mbuf, mb_size,
+ (unsigned int)mb_mempool_cache,
+ sizeof(struct rte_pktmbuf_pool_private),
+ socket_id, mempool_flags);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
if (rte_mp == NULL)
goto err;
@@ -1021,9 +1036,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size,
- heap_socket);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, heap_socket);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_XBUF:
@@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2503,21 +2528,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2618,15 +2650,18 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- if (rte_eth_dev_start(pi) < 0) {
- printf("Fail to start port %d\n", pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d\n", pi);
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to "
"stopped\n", pi);
- continue;
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2755,7 +2790,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2824,8 +2860,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3089,7 +3127,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3611,6 +3649,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3709,13 +3751,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3746,12 +3781,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3776,10 +3815,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3875,8 +3910,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3922,6 +3959,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3957,6 +3996,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index ce83f31..6d13417 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+uint8_t f_quit;
+int testpmd_fd_copy;
+struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -422,6 +427,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 6745072..8e27fcd 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -536,3 +536,72 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support multi process).
+They must be separated from the EAL options, shown in the previous section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the same in primary process
+ and secondary process.
+
+* ``--proc-type``
+ Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
+
+* ``-l CORELIST``
+ List of cores to run on. the corelist should be different in primary process and secondary process.
+
+* ``--rxq=N``
+
+ Set the number of RX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--txq=N``
+
+ Set the number of TX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+Most dev ops is supported in primary and secondary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows:
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
+RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v3] app/testpmd: support multi-process
2021-03-11 9:07 ` [dpdk-dev] [PATCH v3] " Min Hu (Connor)
@ 2021-03-20 0:58 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-20 0:58 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit
Hi, ferruh and all,
have you any comments about this patch?
thanks.
在 2021/3/11 17:07, Min Hu (Connor) 写道:
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> app/test-pmd/cmdline.c | 12 ++-
> app/test-pmd/config.c | 9 ++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
> app/test-pmd/testpmd.h | 7 ++
> doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
> 6 files changed, 196 insertions(+), 50 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index 14110eb..287d7a0 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -71,8 +71,6 @@
> #include "cmdline_tm.h"
> #include "bpf_cmd.h"
>
> -static struct cmdline *testpmd_cl;
> -
> static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
>
> /* *** Help command with introduction. *** */
> @@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
> __rte_unused void *data)
> {
> struct cmd_set_flush_rx *res = parsed_result;
> +
> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> + printf("multi-process doesn't support to flush rx queues.\n");
> + return;
> + }
> +
> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
> }
>
> @@ -17227,6 +17231,10 @@ prompt(void)
> printf("Cannot set exit function for cmdline\n");
>
> cmdline_interact(testpmd_cl);
> + if (unlikely(f_quit == 1)) {
> + dup2(testpmd_fd_copy, testpmd_cl->s_in);
> + close(testpmd_fd_copy);
> + }
> if (ret != 0)
> cmdline_stdin_exit(testpmd_cl);
> }
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index 576d5ac..9edf84a 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
> queueid_t rxq;
> queueid_t nb_q;
> streamid_t sm_id;
> + int start;
> + int end;
>
> nb_q = nb_rxq;
> if (nb_q > nb_txq)
> @@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
> init_fwd_streams();
>
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index c8acd5d..1a5bf25 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -45,6 +45,8 @@
> #include <rte_flow.h>
>
> #include "testpmd.h"
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
>
> static void
> usage(char* progname)
> @@ -605,6 +607,8 @@ launch_args_parse(int argc, char** argv)
> { "rx-mq-mode", 1, 0, 0 },
> { "record-core-cycles", 0, 0, 0 },
> { "record-burst-stats", 0, 0, 0 },
> + { PARAM_NUM_PROCS, 1, 0, 0 },
> + { PARAM_PROC_ID, 1, 0, 0 },
> { 0, 0, 0, 0 },
> };
>
> @@ -1366,6 +1370,13 @@ launch_args_parse(int argc, char** argv)
> record_core_cycles = 1;
> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
> record_burst_stats = 1;
> +
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_NUM_PROCS, 8) == 0)
> + num_procs = atoi(optarg);
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_PROC_ID, 7) == 0)
> + proc_id = atoi(optarg);
> break;
> case 'h':
> usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 1a57324..bbd45d9 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -63,6 +63,8 @@
>
> #include "testpmd.h"
>
> +int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */
> +
> #ifndef MAP_HUGETLB
> /* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
> #define HUGE_FLAG (0x40000)
> @@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
> */
> uint8_t rxring_numa[RTE_MAX_ETHPORTS];
>
> +int proc_id;
> +unsigned int num_procs = 1;
> +
> /*
> * Store specified sockets on which TX ring to be used by ports
> * is allocated.
> @@ -982,16 +987,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
> /* wrapper to rte_mempool_create() */
> TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
> rte_mbuf_best_mempool_ops());
> - rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
> - mb_mempool_cache, 0, mbuf_seg_size, socket_id);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_mp = rte_pktmbuf_pool_create(pool_name,
> + nb_mbuf, mb_mempool_cache, 0,
> + mbuf_seg_size, socket_id);
> + else
> + rte_mp = rte_mempool_lookup(pool_name);
> +
> break;
> }
> case MP_ALLOC_ANON:
> {
> - rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf,
> - mb_size, (unsigned int) mb_mempool_cache,
> - sizeof(struct rte_pktmbuf_pool_private),
> - socket_id, mempool_flags);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_mp = rte_mempool_create_empty(pool_name,
> + nb_mbuf, mb_size,
> + (unsigned int)mb_mempool_cache,
> + sizeof(struct rte_pktmbuf_pool_private),
> + socket_id, mempool_flags);
> + else
> + rte_mp = rte_mempool_lookup(pool_name);
> +
> if (rte_mp == NULL)
> goto err;
>
> @@ -1021,9 +1036,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
>
> TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
> rte_mbuf_best_mempool_ops());
> - rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
> - mb_mempool_cache, 0, mbuf_seg_size,
> - heap_socket);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_mp = rte_pktmbuf_pool_create(pool_name,
> + nb_mbuf, mb_mempool_cache, 0,
> + mbuf_seg_size, heap_socket);
> + else
> + rte_mp = rte_mempool_lookup(pool_name);
> +
> break;
> }
> case MP_ALLOC_XBUF:
> @@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
> uint64_t timer_period;
>
> + if (num_procs > 1) {
> + printf("multi-process not support for flushing fwd rx "
> + "queues, skip the below lines and return.\n");
> + return;
> + }
> +
> /* convert to number of cycles */
> timer_period = rte_get_timer_hz(); /* 1 second timeout */
>
> @@ -2503,21 +2528,28 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_configure(pi,
> + nb_rxq + nb_hairpinq,
> nb_txq + nb_hairpinq,
> &(port->dev_conf));
> - if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> - printf("Fail to configure port %d\n", pi);
> - /* try to reconfigure port next time */
> - port->need_reconfig = 1;
> - return -1;
> + if (diag != 0) {
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set "
> + "back to stopped\n", pi);
> + printf("Fail to configure port %d\n",
> + pi);
> + /* try to reconfigure port next time */
> + port->need_reconfig = 1;
> + return -1;
> + }
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 &&
> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2618,15 +2650,18 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - if (rte_eth_dev_start(pi) < 0) {
> - printf("Fail to start port %d\n", pi);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_start(pi);
> + if (diag < 0) {
> + printf("Fail to start port %d\n", pi);
>
> - /* Fail to setup rx queue, return */
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> + /* Fail to setup rx queue, return */
> + if (rte_atomic16_cmpset(&(port->port_status),
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set back to "
> "stopped\n", pi);
> - continue;
> + continue;
> + }
> }
>
> if (rte_atomic16_cmpset(&(port->port_status),
> @@ -2755,7 +2790,8 @@ stop_port(portid_t pid)
> if (port->flow_list)
> port_flow_flush(pi);
>
> - if (rte_eth_dev_stop(pi) != 0)
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
> + rte_eth_dev_stop(pi) != 0)
> RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
> pi);
>
> @@ -2824,8 +2860,10 @@ close_port(portid_t pid)
> continue;
> }
>
> - port_flow_flush(pi);
> - rte_eth_dev_close(pi);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + port_flow_flush(pi);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_eth_dev_close(pi);
> }
>
> remove_invalid_ports();
> @@ -3089,7 +3127,7 @@ pmd_test_exit(void)
> }
> }
> for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
> - if (mempools[i])
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
> rte_mempool_free(mempools[i]);
> }
>
> @@ -3611,6 +3649,10 @@ init_port_dcb_config(portid_t pid,
> int retval;
> uint16_t i;
>
> + if (num_procs > 1) {
> + printf("The multi-process feature doesn't support dcb.\n");
> + return -ENOTSUP;
> + }
> rte_port = &ports[pid];
>
> memset(&port_conf, 0, sizeof(struct rte_eth_conf));
> @@ -3709,13 +3751,6 @@ init_port(void)
> }
>
> static void
> -force_quit(void)
> -{
> - pmd_test_exit();
> - prompt_exit();
> -}
> -
> -static void
> print_stats(void)
> {
> uint8_t i;
> @@ -3746,12 +3781,16 @@ signal_handler(int signum)
> if (latencystats_enabled != 0)
> rte_latencystats_uninit();
> #endif
> - force_quit();
> /* Set flag to indicate the force termination. */
> f_quit = 1;
> - /* exit with the expected status */
> - signal(signum, SIG_DFL);
> - kill(getpid(), signum);
> + if (interactive == 1) {
> + dup2(testpmd_cl->s_in, testpmd_fd_copy);
> + close(testpmd_cl->s_in);
> + } else {
> + dup2(0, testpmd_fd_copy);
> + close(0);
> + }
> +
> }
> }
>
> @@ -3776,10 +3815,6 @@ main(int argc, char** argv)
> rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
> rte_strerror(rte_errno));
>
> - if (rte_eal_process_type() == RTE_PROC_SECONDARY)
> - rte_exit(EXIT_FAILURE,
> - "Secondary process type not supported.\n");
> -
> ret = register_eth_event_callback();
> if (ret != 0)
> rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
> @@ -3875,8 +3910,10 @@ main(int argc, char** argv)
> }
> }
>
> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
> + pmd_test_exit();
> rte_exit(EXIT_FAILURE, "Start ports failed\n");
> + }
>
> /* set all ports to promiscuous mode by default */
> RTE_ETH_FOREACH_DEV(port_id) {
> @@ -3922,6 +3959,8 @@ main(int argc, char** argv)
> }
> prompt();
> pmd_test_exit();
> + if (unlikely(f_quit == 1))
> + prompt_exit();
> } else
> #endif
> {
> @@ -3957,6 +3996,11 @@ main(int argc, char** argv)
> printf("Press enter to exit\n");
> rc = read(0, &c, 1);
> pmd_test_exit();
> + if (unlikely(f_quit == 1)) {
> + dup2(testpmd_fd_copy, 0);
> + close(testpmd_fd_copy);
> + prompt_exit();
> + }
> if (rc < 0)
> return 1;
> }
> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
> index ce83f31..6d13417 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
> @@ -13,6 +13,7 @@
> #include <rte_gso.h>
> #include <cmdline.h>
> #include <sys/queue.h>
> +#include "cmdline.h"
>
> #define RTE_PORT_ALL (~(portid_t)0x0)
>
> @@ -24,6 +25,10 @@
> #define RTE_PORT_CLOSED (uint16_t)2
> #define RTE_PORT_HANDLING (uint16_t)3
>
> +uint8_t f_quit;
> +int testpmd_fd_copy;
> +struct cmdline *testpmd_cl;
> +
> /*
> * It is used to allocate the memory for hash key.
> * The hash key size is NIC dependent.
> @@ -422,6 +427,8 @@ extern uint64_t noisy_lkup_mem_sz;
> extern uint64_t noisy_lkup_num_writes;
> extern uint64_t noisy_lkup_num_reads;
> extern uint64_t noisy_lkup_num_reads_writes;
> +extern int proc_id;
> +extern unsigned int num_procs;
>
> extern uint8_t dcb_config;
> extern uint8_t dcb_test;
> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
> index 6745072..8e27fcd 100644
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -536,3 +536,72 @@ The command line options are:
> bit 1 - two hairpin ports paired
> bit 0 - two hairpin ports loop
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +
> +Testpmd Support Multi Process Command-line Options
> +--------------------------------------------------
> +
> +The following are the command-line options for the testpmd applications(support multi process).
> +They must be separated from the EAL options, shown in the previous section, with a ``--`` separator:
> +
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=1
> +
> +The command line options are:
> +
> +* ``-a, --allow``
> +
> + Add a device to the allow list. ``xxx`` means device used which should be the same in primary process
> + and secondary process.
> +
> +* ``--proc-type``
> + Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
> +
> +* ``-l CORELIST``
> + List of cores to run on. the corelist should be different in primary process and secondary process.
> +
> +* ``--rxq=N``
> +
> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary process.
> +
> +* ``--txq=N``
> +
> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary process.
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary process and secondary process.
> +
> +Calculation rule for queue:
> +All queues are allocated to different processes based on proc_num and proc_id.
> +Calculation rule for the Testpmd to allocate queues to each process:
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
> +For example, if supports 4 txq and rxq
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +Most dev ops is supported in primary and secondary process. While secondary process is not permitted
> +to allocate or release shared memory, so some ops are not supported as follows:
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
> +stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
> +RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (2 preceding siblings ...)
2021-03-11 9:07 ` [dpdk-dev] [PATCH v3] " Min Hu (Connor)
@ 2021-03-22 7:07 ` Min Hu (Connor)
2021-03-22 11:19 ` Ferruh Yigit
2021-03-24 8:08 ` Li, Xiaoyun
2021-03-25 13:17 ` [dpdk-dev] [PATCH v5] " Min Hu (Connor)
` (8 subsequent siblings)
12 siblings, 2 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-22 7:07 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, ajit.khaparde
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 12 ++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
6 files changed, 196 insertions(+), 50 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 576d5ac..9edf84a 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c8acd5d..1a5bf25 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -605,6 +607,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1366,6 +1370,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1a57324..bbd45d9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,8 @@
#include "testpmd.h"
+int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -982,16 +987,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
/* wrapper to rte_mempool_create() */
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size, socket_id);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, socket_id);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_ANON:
{
- rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf,
- mb_size, (unsigned int) mb_mempool_cache,
- sizeof(struct rte_pktmbuf_pool_private),
- socket_id, mempool_flags);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_mempool_create_empty(pool_name,
+ nb_mbuf, mb_size,
+ (unsigned int)mb_mempool_cache,
+ sizeof(struct rte_pktmbuf_pool_private),
+ socket_id, mempool_flags);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
if (rte_mp == NULL)
goto err;
@@ -1021,9 +1036,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size,
- heap_socket);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, heap_socket);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_XBUF:
@@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2503,21 +2528,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2618,15 +2650,18 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- if (rte_eth_dev_start(pi) < 0) {
- printf("Fail to start port %d\n", pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d\n", pi);
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to "
"stopped\n", pi);
- continue;
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2755,7 +2790,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2824,8 +2860,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3089,7 +3127,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3611,6 +3649,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3709,13 +3751,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3746,12 +3781,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3776,10 +3815,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3875,8 +3910,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3922,6 +3959,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3957,6 +3996,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index ce83f31..6d13417 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+uint8_t f_quit;
+int testpmd_fd_copy;
+struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -422,6 +427,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index 6745072..8e27fcd 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -536,3 +536,72 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support multi process).
+They must be separated from the EAL options, shown in the previous section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the same in primary process
+ and secondary process.
+
+* ``--proc-type``
+ Specify a given process instance as the primary or secondary DPDK instance. ``auto`` set here is ok.
+
+* ``-l CORELIST``
+ List of cores to run on. the corelist should be different in primary process and secondary process.
+
+* ``--rxq=N``
+
+ Set the number of RX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--txq=N``
+
+ Set the number of TX queues per port to N, where 1 <= N <= 65535.
+ The default value is 1. N is the sum of queues used by primary and secondary process.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+Most dev ops is supported in primary and secondary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows:
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same buffer to store the stats.
+RSS supported, Primary process and secondary process has separate queues to use, RSS will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-22 7:07 ` [dpdk-dev] [PATCH v4] " Min Hu (Connor)
@ 2021-03-22 11:19 ` Ferruh Yigit
2021-03-24 8:08 ` Li, Xiaoyun
1 sibling, 0 replies; 80+ messages in thread
From: Ferruh Yigit @ 2021-03-22 11:19 UTC (permalink / raw)
To: Min Hu (Connor), dev, Xiaoyun Li; +Cc: ajit.khaparde
On 3/22/2021 7:07 AM, Min Hu (Connor) wrote:
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
+Xiaoyun, testpmd maintainer.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-22 7:07 ` [dpdk-dev] [PATCH v4] " Min Hu (Connor)
2021-03-22 11:19 ` Ferruh Yigit
@ 2021-03-24 8:08 ` Li, Xiaoyun
2021-03-25 13:32 ` Min Hu (Connor)
1 sibling, 1 reply; 80+ messages in thread
From: Li, Xiaoyun @ 2021-03-24 8:08 UTC (permalink / raw)
To: Min Hu (Connor), dev, ajit.khaparde; +Cc: Yigit, Ferruh
Hi
> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Min Hu (Connor)
> Sent: Monday, March 22, 2021 15:07
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com
> Subject: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
>
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 12 ++-
> app/test-pmd/config.c | 9 ++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
> app/test-pmd/testpmd.h | 7 ++
> doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
> 6 files changed, 196 insertions(+), 50 deletions(-)
>
<snip>
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_mp = rte_pktmbuf_pool_create(pool_name,
> + nb_mbuf, mb_mempool_cache, 0,
> + mbuf_seg_size, heap_socket);
> + else
> + rte_mp = rte_mempool_lookup(pool_name);
> +
> break;
> }
> case MP_ALLOC_XBUF:
What about this one when users use external bufs? Why not addressing secondary process here?
If it works for all cases, you should add a condition at the start of this function, if it's secondary, goto err to check mp and return.
> @@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
> uint64_t timer_period;
>
> + if (num_procs > 1) {
> + printf("multi-process not support for flushing fwd rx "
> + "queues, skip the below lines and return.\n");
<snip>
> +uint8_t f_quit;
> +int testpmd_fd_copy;
> +struct cmdline *testpmd_cl;
> +
Please address the compilation failure on patchwork related to these variables (multiple definitions).
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
> +--txq=4 --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
> +--txq=4 --num-procs=2 --proc-id=1
> +
<snip>
> +* ``--rxq=N``
> +
> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary
> process.
> +
Did you upstream wrong patch?
You said you would address the queue number issue Ajit Khaparde mentioned but you didn't in this patch.
The number of queues should be a multiple of the number of processes?
> +* ``--txq=N``
> +
> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
> + The default value is 1. N is the sum of queues used by primary and secondary
> process.
> +
Same as above.
> +* ``--num-procs=N``
<snip>
> +Most dev ops is supported in primary and secondary process. While
> +secondary process is not permitted to allocate or release shared memory, so
> some ops are not supported as follows:
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
What about some config commands?
Such as "clear port stats all". Should this be allowed by secondary?
And like "port config all rxq". If primary hasn't started ports, should the secondary allowed to change traffic related stuff (offloads, rx/txd, rx/txq and so on)?
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on
> HW size.
About rte flow, what do you mean apply only on its own process on SW side?
If I set number-procs=2, rxq=4
Then on secondary process, I set a flow which directs 192.168.0.1 traffic to queue 0. It seems it will directs this kind of traffic to primary process. But I can't see this rule from primary process side.
Is this behavior right for multiple process?
> +stats supported, stats will not change when one quit and start, As they share
> the same buffer to store the stats.
> +RSS supported, Primary process and secondary process has separate queues to
> use, RSS will work in their own queues whether primary and secondary process.
> --
> 2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-24 8:08 ` Li, Xiaoyun
@ 2021-03-25 13:32 ` Min Hu (Connor)
2021-03-25 23:25 ` Ajit Khaparde
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-25 13:32 UTC (permalink / raw)
To: Li, Xiaoyun, dev, ajit.khaparde; +Cc: Yigit, Ferruh
在 2021/3/24 16:08, Li, Xiaoyun 写道:
> Hi
>
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Min Hu (Connor)
>> Sent: Monday, March 22, 2021 15:07
>> To: dev@dpdk.org
>> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com
>> Subject: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
>>
>> From: Lijun Ou <oulijun@huawei.com>
>>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> ---
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 12 ++-
>> app/test-pmd/config.c | 9 ++-
>> app/test-pmd/parameters.c | 11 +++
>> app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
>> app/test-pmd/testpmd.h | 7 ++
>> doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
>> 6 files changed, 196 insertions(+), 50 deletions(-)
>>
> <snip>
>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
>> + rte_mp = rte_pktmbuf_pool_create(pool_name,
>> + nb_mbuf, mb_mempool_cache, 0,
>> + mbuf_seg_size, heap_socket);
>> + else
>> + rte_mp = rte_mempool_lookup(pool_name);
>> +
>> break;
>> }
>> case MP_ALLOC_XBUF:
>
> What about this one when users use external bufs? Why not addressing secondary process here?
> If it works for all cases, you should add a condition at the start of this function, if it's secondary, goto err to check mp and return.
>
Yes, your are right, I have fixed it in v5, thanks.
>> @@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
>> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
>> uint64_t timer_period;
>>
>> + if (num_procs > 1) {
>> + printf("multi-process not support for flushing fwd rx "
>> + "queues, skip the below lines and return.\n");
>
> <snip>
>> +uint8_t f_quit;
>> +int testpmd_fd_copy;
>> +struct cmdline *testpmd_cl;
>> +
>
> Please address the compilation failure on patchwork related to these variables (multiple definitions).
>
Done in v5.
>> +.. code-block:: console
>> +
>> + primary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
>> +--txq=4 --num-procs=2 --proc-id=0
>> +
>> + secondary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
>> +--txq=4 --num-procs=2 --proc-id=1
>> +
> <snip>
>> +* ``--rxq=N``
>> +
>> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
>> + The default value is 1. N is the sum of queues used by primary and secondary
>> process.
>> +
>
> Did you upstream wrong patch?
> You said you would address the queue number issue Ajit Khaparde mentioned but you didn't in this patch.
> The number of queues should be a multiple of the number of processes?
>
Done in v5.
>> +* ``--txq=N``
>> +
>> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
>> + The default value is 1. N is the sum of queues used by primary and secondary
>> process.
>> +
> Same as above.
>
>> +* ``--num-procs=N``
> <snip>
>> +Most dev ops is supported in primary and secondary process. While
>> +secondary process is not permitted to allocate or release shared memory, so
>> some ops are not supported as follows:
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
>
> What about some config commands?
> Such as "clear port stats all". Should this be allowed by secondary?
>
I think so, actually, all the queues is visible to primary and
secondary. The only thing we do is to separate queues for different
process for io (packets) in Rx/Tx. It is of for secondary "clear port
stats all".
> And like "port config all rxq". If primary hasn't started ports, should the secondary allowed to change traffic related stuff (offloads, rx/txd, rx/txq and so on)?
>
Yes, port config all rxq/txq/rxd/txd/offload is not supported in the
secondary process. It has been done in v5.
>> +
>> +RTE_FLOW supported, it applies only on its own process on SW side, but all on
>> HW size.
>
> About rte flow, what do you mean apply only on its own process on SW side?
> If I set number-procs=2, rxq=4
> Then on secondary process, I set a flow which directs 192.168.0.1 traffic to queue 0. It seems it will directs this kind of traffic to primary process. But I can't see this rule from primary process side.
> Is this behavior right for multiple process?
>
According to doc rte_flow.rst, we maintain flow rules in process level:
primary and secondary has its own flow list(but one flow list in HW).
As previously mentioned, the two can see all the queues, so setting the
flow rules for the other is OK.
Of course, io(receive or transmit packets) in the queue in others is not
permitted.
>> +stats supported, stats will not change when one quit and start, As they share
>> the same buffer to store the stats.
>> +RSS supported, Primary process and secondary process has separate queues to
>> use, RSS will work in their own queues whether primary and secondary process.
>> --
>> 2.7.4
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-25 13:32 ` Min Hu (Connor)
@ 2021-03-25 23:25 ` Ajit Khaparde
2021-03-26 6:46 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ajit Khaparde @ 2021-03-25 23:25 UTC (permalink / raw)
To: Min Hu (Connor); +Cc: Li, Xiaoyun, dev, Yigit, Ferruh
[-- Attachment #1: Type: text/plain, Size: 6515 bytes --]
On Thu, Mar 25, 2021 at 6:32 AM Min Hu (Connor) <humin29@huawei.com> wrote:
>
>
>
> 在 2021/3/24 16:08, Li, Xiaoyun 写道:
> > Hi
> >
> >> -----Original Message-----
> >> From: dev <dev-bounces@dpdk.org> On Behalf Of Min Hu (Connor)
> >> Sent: Monday, March 22, 2021 15:07
> >> To: dev@dpdk.org
> >> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com
> >> Subject: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
> >>
> >> From: Lijun Ou <oulijun@huawei.com>
> >>
> >> This patch adds multi-process support for testpmd.
> >> The test cmd example as follows:
> >> the primary cmd:
> >> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> >> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
> >>
> >> the secondary cmd:
> >> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> >> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
> >>
> >> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> >> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> >> ---
> >> v4:
> >> * Fixed minimum vlaue of Rxq or Txq in doc.
> >>
> >> v3:
> >> * Fixed compiling error using gcc10.0.
> >>
> >> v2:
> >> * Added document for this patch.
> >> ---
> >> app/test-pmd/cmdline.c | 12 ++-
> >> app/test-pmd/config.c | 9 ++-
> >> app/test-pmd/parameters.c | 11 +++
> >> app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
> >> app/test-pmd/testpmd.h | 7 ++
> >> doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
> >> 6 files changed, 196 insertions(+), 50 deletions(-)
> >>
> > <snip>
> >> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> >> + rte_mp = rte_pktmbuf_pool_create(pool_name,
> >> + nb_mbuf, mb_mempool_cache, 0,
> >> + mbuf_seg_size, heap_socket);
> >> + else
> >> + rte_mp = rte_mempool_lookup(pool_name);
> >> +
> >> break;
> >> }
> >> case MP_ALLOC_XBUF:
> >
> > What about this one when users use external bufs? Why not addressing secondary process here?
> > If it works for all cases, you should add a condition at the start of this function, if it's secondary, goto err to check mp and return.
> >
> Yes, your are right, I have fixed it in v5, thanks.
> >> @@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
> >> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
> >> uint64_t timer_period;
> >>
> >> + if (num_procs > 1) {
> >> + printf("multi-process not support for flushing fwd rx "
> >> + "queues, skip the below lines and return.\n");
> >
> > <snip>
> >> +uint8_t f_quit;
> >> +int testpmd_fd_copy;
> >> +struct cmdline *testpmd_cl;
> >> +
> >
> > Please address the compilation failure on patchwork related to these variables (multiple definitions).
> >
> Done in v5.
> >> +.. code-block:: console
> >> +
> >> + primary process:
> >> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
> >> +--txq=4 --num-procs=2 --proc-id=0
> >> +
> >> + secondary process:
> >> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
> >> +--txq=4 --num-procs=2 --proc-id=1
> >> +
> > <snip>
> >> +* ``--rxq=N``
> >> +
> >> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
> >> + The default value is 1. N is the sum of queues used by primary and secondary
> >> process.
> >> +
> >
> > Did you upstream wrong patch?
> > You said you would address the queue number issue Ajit Khaparde mentioned but you didn't in this patch.
> > The number of queues should be a multiple of the number of processes?
> >
> Done in v5.
> >> +* ``--txq=N``
> >> +
> >> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
> >> + The default value is 1. N is the sum of queues used by primary and secondary
> >> process.
> >> +
> > Same as above.
> >
> >> +* ``--num-procs=N``
> > <snip>
> >> +Most dev ops is supported in primary and secondary process. While
> >> +secondary process is not permitted to allocate or release shared memory, so
> >> some ops are not supported as follows:
> >> +``dev_start``
> >> +``dev_stop``
> >> +``rx_queue_setup``
> >> +``tx_queue_setup``
> >> +``rx_queue_release``
> >> +``tx_queue_release``
> >
> > What about some config commands?
> > Such as "clear port stats all". Should this be allowed by secondary?
> >
> I think so, actually, all the queues is visible to primary and
> secondary. The only thing we do is to separate queues for different
> process for io (packets) in Rx/Tx. It is of for secondary "clear port
> stats all".
> > And like "port config all rxq". If primary hasn't started ports, should the secondary allowed to change traffic related stuff (offloads, rx/txd, rx/txq and so on)?
> >
> Yes, port config all rxq/txq/rxd/txd/offload is not supported in the
> secondary process. It has been done in v5.
> >> +
> >> +RTE_FLOW supported, it applies only on its own process on SW side, but all on
> >> HW size.
> >
> > About rte flow, what do you mean apply only on its own process on SW side?
> > If I set number-procs=2, rxq=4
> > Then on secondary process, I set a flow which directs 192.168.0.1 traffic to queue 0. It seems it will directs this kind of traffic to primary process. But I can't see this rule from primary process side.
> > Is this behavior right for multiple process?
> >
> According to doc rte_flow.rst, we maintain flow rules in process level:
> primary and secondary has its own flow list(but one flow list in HW).
> As previously mentioned, the two can see all the queues, so setting the
> flow rules for the other is OK.
> Of course, io(receive or transmit packets) in the queue in others is not
> permitted.
Can you add this behavior as well to the testpmd doc.
Further isolation of resources and operations between the primary and
secondary processes is possible.
But this is a good start. We can add more if needed.
> >> +stats supported, stats will not change when one quit and start, As they share
> >> the same buffer to store the stats.
> >> +RSS supported, Primary process and secondary process has separate queues to
> >> use, RSS will work in their own queues whether primary and secondary process.
> >> --
> >> 2.7.4
> >
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
2021-03-25 23:25 ` Ajit Khaparde
@ 2021-03-26 6:46 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-26 6:46 UTC (permalink / raw)
To: Ajit Khaparde; +Cc: Li, Xiaoyun, dev, Yigit, Ferruh
在 2021/3/26 7:25, Ajit Khaparde 写道:
> On Thu, Mar 25, 2021 at 6:32 AM Min Hu (Connor) <humin29@huawei.com> wrote:
>>
>>
>>
>> 在 2021/3/24 16:08, Li, Xiaoyun 写道:
>>> Hi
>>>
>>>> -----Original Message-----
>>>> From: dev <dev-bounces@dpdk.org> On Behalf Of Min Hu (Connor)
>>>> Sent: Monday, March 22, 2021 15:07
>>>> To: dev@dpdk.org
>>>> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com
>>>> Subject: [dpdk-dev] [PATCH v4] app/testpmd: support multi-process
>>>>
>>>> From: Lijun Ou <oulijun@huawei.com>
>>>>
>>>> This patch adds multi-process support for testpmd.
>>>> The test cmd example as follows:
>>>> the primary cmd:
>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>
>>>> the secondary cmd:
>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>
>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>> ---
>>>> v4:
>>>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>>>
>>>> v3:
>>>> * Fixed compiling error using gcc10.0.
>>>>
>>>> v2:
>>>> * Added document for this patch.
>>>> ---
>>>> app/test-pmd/cmdline.c | 12 ++-
>>>> app/test-pmd/config.c | 9 ++-
>>>> app/test-pmd/parameters.c | 11 +++
>>>> app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++------------
>>>> app/test-pmd/testpmd.h | 7 ++
>>>> doc/guides/testpmd_app_ug/run_app.rst | 69 +++++++++++++++++
>>>> 6 files changed, 196 insertions(+), 50 deletions(-)
>>>>
>>> <snip>
>>>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
>>>> + rte_mp = rte_pktmbuf_pool_create(pool_name,
>>>> + nb_mbuf, mb_mempool_cache, 0,
>>>> + mbuf_seg_size, heap_socket);
>>>> + else
>>>> + rte_mp = rte_mempool_lookup(pool_name);
>>>> +
>>>> break;
>>>> }
>>>> case MP_ALLOC_XBUF:
>>>
>>> What about this one when users use external bufs? Why not addressing secondary process here?
>>> If it works for all cases, you should add a condition at the start of this function, if it's secondary, goto err to check mp and return.
>>>
>> Yes, your are right, I have fixed it in v5, thanks.
>>>> @@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
>>>> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
>>>> uint64_t timer_period;
>>>>
>>>> + if (num_procs > 1) {
>>>> + printf("multi-process not support for flushing fwd rx "
>>>> + "queues, skip the below lines and return.\n");
>>>
>>> <snip>
>>>> +uint8_t f_quit;
>>>> +int testpmd_fd_copy;
>>>> +struct cmdline *testpmd_cl;
>>>> +
>>>
>>> Please address the compilation failure on patchwork related to these variables (multiple definitions).
>>>
>> Done in v5.
>>>> +.. code-block:: console
>>>> +
>>>> + primary process:
>>>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
>>>> +--txq=4 --num-procs=2 --proc-id=0
>>>> +
>>>> + secondary process:
>>>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
>>>> +--txq=4 --num-procs=2 --proc-id=1
>>>> +
>>> <snip>
>>>> +* ``--rxq=N``
>>>> +
>>>> + Set the number of RX queues per port to N, where 1 <= N <= 65535.
>>>> + The default value is 1. N is the sum of queues used by primary and secondary
>>>> process.
>>>> +
>>>
>>> Did you upstream wrong patch?
>>> You said you would address the queue number issue Ajit Khaparde mentioned but you didn't in this patch.
>>> The number of queues should be a multiple of the number of processes?
>>>
>> Done in v5.
>>>> +* ``--txq=N``
>>>> +
>>>> + Set the number of TX queues per port to N, where 1 <= N <= 65535.
>>>> + The default value is 1. N is the sum of queues used by primary and secondary
>>>> process.
>>>> +
>>> Same as above.
>>>
>>>> +* ``--num-procs=N``
>>> <snip>
>>>> +Most dev ops is supported in primary and secondary process. While
>>>> +secondary process is not permitted to allocate or release shared memory, so
>>>> some ops are not supported as follows:
>>>> +``dev_start``
>>>> +``dev_stop``
>>>> +``rx_queue_setup``
>>>> +``tx_queue_setup``
>>>> +``rx_queue_release``
>>>> +``tx_queue_release``
>>>
>>> What about some config commands?
>>> Such as "clear port stats all". Should this be allowed by secondary?
>> >
>> I think so, actually, all the queues is visible to primary and
>> secondary. The only thing we do is to separate queues for different
>> process for io (packets) in Rx/Tx. It is of for secondary "clear port
>> stats all".
>>> And like "port config all rxq". If primary hasn't started ports, should the secondary allowed to change traffic related stuff (offloads, rx/txd, rx/txq and so on)?
>>>
>> Yes, port config all rxq/txq/rxd/txd/offload is not supported in the
>> secondary process. It has been done in v5.
>>>> +
>>>> +RTE_FLOW supported, it applies only on its own process on SW side, but all on
>>>> HW size.
>>>
>>> About rte flow, what do you mean apply only on its own process on SW side?
>>> If I set number-procs=2, rxq=4
>>> Then on secondary process, I set a flow which directs 192.168.0.1 traffic to queue 0. It seems it will directs this kind of traffic to primary process. But I can't see this rule from primary process side.
>>> Is this behavior right for multiple process?
>>>
>> According to doc rte_flow.rst, we maintain flow rules in process level:
>> primary and secondary has its own flow list(but one flow list in HW).
>> As previously mentioned, the two can see all the queues, so setting the
>> flow rules for the other is OK.
>> Of course, io(receive or transmit packets) in the queue in others is not
>> permitted.
> Can you add this behavior as well to the testpmd doc.
>
Hi, Ajit, done in v6, thanks.
> Further isolation of resources and operations between the primary and
> secondary processes is possible.
> But this is a good start. We can add more if needed.
>
>>>> +stats supported, stats will not change when one quit and start, As they share
>>>> the same buffer to store the stats.
>>>> +RSS supported, Primary process and secondary process has separate queues to
>>>> use, RSS will work in their own queues whether primary and secondary process.
>>>> --
>>>> 2.7.4
>>>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v5] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (3 preceding siblings ...)
2021-03-22 7:07 ` [dpdk-dev] [PATCH v4] " Min Hu (Connor)
@ 2021-03-25 13:17 ` Min Hu (Connor)
2021-03-26 6:46 ` [dpdk-dev] [PATCH v6] " Min Hu (Connor)
` (7 subsequent siblings)
12 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-25 13:17 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, xiaoyun.li
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 12 +++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 95 +++++++++++++++++++++++++
6 files changed, 215 insertions(+), 46 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ef0b978..e48cbd9 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index a326c8c..ec3bc62 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -644,6 +646,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1410,6 +1414,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..c31234e 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,9 @@
#include "testpmd.h"
+int testpmd_fd_copy; /* the copy of STDIN_FILENO */
+struct cmdline *testpmd_cl;
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +128,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -977,6 +983,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1070,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2018,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2533,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2655,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2797,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2867,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3099,7 +3134,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3621,6 +3656,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3719,13 +3758,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3756,12 +3788,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3786,10 +3822,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3917,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3932,6 +3966,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3967,6 +4003,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index a87ccb0..640a377 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+extern uint8_t f_quit;
+extern int testpmd_fd_copy;
+extern struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -423,6 +428,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..b35e6d9 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,98 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the
+ same in primary process and secondary process.
+
+* ``--proc-type``
+
+ Specify a given process instance as the primary or secondary DPDK instance.
+ ``auto`` set here is OK.
+
+* ``-l CORELIST``
+
+ List of cores to run on. the corelist should be different in primary process and
+ secondary process.
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings had better be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats.
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* [dpdk-dev] [PATCH v6] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (4 preceding siblings ...)
2021-03-25 13:17 ` [dpdk-dev] [PATCH v5] " Min Hu (Connor)
@ 2021-03-26 6:46 ` Min Hu (Connor)
2021-03-26 8:52 ` [dpdk-dev] [PATCH v7] " Min Hu (Connor)
` (6 subsequent siblings)
12 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-26 6:46 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, ajit.khaparde, xiaoyun.li
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 12 +++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 99 ++++++++++++++++++++++++++
6 files changed, 219 insertions(+), 46 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ef0b978..e48cbd9 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index a326c8c..ec3bc62 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -644,6 +646,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1410,6 +1414,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..c31234e 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,9 @@
#include "testpmd.h"
+int testpmd_fd_copy; /* the copy of STDIN_FILENO */
+struct cmdline *testpmd_cl;
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +128,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -977,6 +983,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1070,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2018,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2533,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2655,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2797,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2867,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3099,7 +3134,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3621,6 +3656,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3719,13 +3758,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3756,12 +3788,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3786,10 +3822,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3917,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3932,6 +3966,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3967,6 +4003,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index a87ccb0..640a377 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+extern uint8_t f_quit;
+extern int testpmd_fd_copy;
+extern struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -423,6 +428,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..a59f2aa 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,102 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the
+ same in primary process and secondary process.
+
+* ``--proc-type``
+
+ Specify a given process instance as the primary or secondary DPDK instance.
+ ``auto`` set here is OK.
+
+* ``-l CORELIST``
+
+ List of cores to run on. the corelist should be different in primary process and
+ secondary process.
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings had better be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* [dpdk-dev] [PATCH v7] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (5 preceding siblings ...)
2021-03-26 6:46 ` [dpdk-dev] [PATCH v6] " Min Hu (Connor)
@ 2021-03-26 8:52 ` Min Hu (Connor)
2021-03-29 7:51 ` Li, Xiaoyun
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
` (5 subsequent siblings)
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-26 8:52 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, ajit.khaparde, xiaoyun.li
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 12 +++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 99 ++++++++++++++++++++++++++
6 files changed, 219 insertions(+), 46 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 14110eb..287d7a0 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17227,6 +17231,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ef0b978..e48cbd9 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index a326c8c..ec3bc62 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -644,6 +646,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1410,6 +1414,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..c31234e 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,9 @@
#include "testpmd.h"
+int testpmd_fd_copy; /* the copy of STDIN_FILENO */
+struct cmdline *testpmd_cl;
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +128,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -977,6 +983,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1070,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2018,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2533,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2655,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2797,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2867,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3099,7 +3134,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3621,6 +3656,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3719,13 +3758,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3756,12 +3788,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3786,10 +3822,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3917,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3932,6 +3966,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3967,6 +4003,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index a87ccb0..640a377 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+extern uint8_t f_quit;
+extern int testpmd_fd_copy;
+extern struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -423,6 +428,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..ce85aa0 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,102 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the
+ same in primary process and secondary process.
+
+* ``--proc-type``
+
+ Specify a given process instance as the primary or secondary DPDK instance.
+ ``auto`` set here is OK.
+
+* ``-l CORELIST``
+
+ List of cores to run on. the corelist should be different in primary process and
+ secondary process.
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. As primary process and secondary process should have separate
+ queues, and each should occupy at least one queue.where N should be no less than two.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings had better be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v7] app/testpmd: support multi-process
2021-03-26 8:52 ` [dpdk-dev] [PATCH v7] " Min Hu (Connor)
@ 2021-03-29 7:51 ` Li, Xiaoyun
2021-03-30 1:48 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Li, Xiaoyun @ 2021-03-29 7:51 UTC (permalink / raw)
To: Min Hu (Connor), dev; +Cc: Yigit, Ferruh, ajit.khaparde
Hi
> -----Original Message-----
> From: Min Hu (Connor) <humin29@huawei.com>
> Sent: Friday, March 26, 2021 16:53
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com; Li,
> Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH v7] app/testpmd: support multi-process
>
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 12 +++-
> app/test-pmd/config.c | 9 ++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
> app/test-pmd/testpmd.h | 7 ++
> doc/guides/testpmd_app_ug/run_app.rst | 99 ++++++++++++++++++++++++++
> 6 files changed, 219 insertions(+), 46 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
> 14110eb..287d7a0 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -71,8 +71,6 @@
Why not just restrict queue numbers to be multiple of processes here in rss_fwd_config_setup()?
Or at least print a warning to tell users if it's not multiple, packet loss will happen.
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> +* ``--txq=N``
> +
> + Set the number of Tx queues per port to N. N is the sum of queues used by
> primary
> + and secondary process. As primary process and secondary process should
> have separate
> + queues, and each should occupy at least one queue.where N should be no
> less than two.
Where N should be the multiple of number of processes.
BRs
Xiaoyun
> +
> 2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v7] app/testpmd: support multi-process
2021-03-29 7:51 ` Li, Xiaoyun
@ 2021-03-30 1:48 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-30 1:48 UTC (permalink / raw)
To: Li, Xiaoyun, dev; +Cc: Yigit, Ferruh, ajit.khaparde
在 2021/3/29 15:51, Li, Xiaoyun 写道:
> Hi
>
>> -----Original Message-----
>> From: Min Hu (Connor) <humin29@huawei.com>
>> Sent: Friday, March 26, 2021 16:53
>> To: dev@dpdk.org
>> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com; Li,
>> Xiaoyun <xiaoyun.li@intel.com>
>> Subject: [PATCH v7] app/testpmd: support multi-process
>>
>> From: Lijun Ou <oulijun@huawei.com>
>>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> ---
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 12 +++-
>> app/test-pmd/config.c | 9 ++-
>> app/test-pmd/parameters.c | 11 +++
>> app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
>> app/test-pmd/testpmd.h | 7 ++
>> doc/guides/testpmd_app_ug/run_app.rst | 99 ++++++++++++++++++++++++++
>> 6 files changed, 219 insertions(+), 46 deletions(-)
>>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index
>> 14110eb..287d7a0 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -71,8 +71,6 @@
>
> Why not just restrict queue numbers to be multiple of processes here in rss_fwd_config_setup()?
> Or at least print a warning to tell users if it's not multiple, packet loss will happen.
>
Hi, fixed in v8, thanks.
>> setup_fwd_config_of_each_lcore(&cur_fwd_config);
>> - rxp = 0; rxq = 0;
>> + start = proc_id * nb_q / num_procs;
>> + end = start + nb_q / num_procs;
>> + rxp = 0;
>> + rxq = start;
>> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>> struct fwd_stream *fs;
>>
>> @@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
>> continue;
>> rxp = 0;
>> rxq++;
>> + if (rxq >= end)
>> + rxq = start;
>> }
>> }
>>
>> +* ``--txq=N``
>> +
>> + Set the number of Tx queues per port to N. N is the sum of queues used by
>> primary
>> + and secondary process. As primary process and secondary process should
>> have separate
>> + queues, and each should occupy at least one queue.where N should be no
>> less than two.
>
> Where N should be the multiple of number of processes.
>
Fixed in v8, thanks.
> BRs
> Xiaoyun
>> +
>> 2.7.4
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (6 preceding siblings ...)
2021-03-26 8:52 ` [dpdk-dev] [PATCH v7] " Min Hu (Connor)
@ 2021-03-30 1:48 ` Min Hu (Connor)
2021-03-30 2:17 ` Li, Xiaoyun
` (2 more replies)
2021-04-16 1:52 ` [dpdk-dev] [PATCH v9] " Min Hu (Connor)
` (4 subsequent siblings)
12 siblings, 3 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-30 1:48 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, ajit.khaparde, xiaoyun.li
From: Lijun Ou <oulijun@huawei.com>
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 12 +++-
app/test-pmd/config.c | 14 +++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
app/test-pmd/testpmd.h | 7 ++
doc/guides/testpmd_app_ug/run_app.rst | 101 +++++++++++++++++++++++++++
6 files changed, 226 insertions(+), 46 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index f44116b..38c97e3 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -71,8 +71,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17229,6 +17233,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ef0b978..ec7d030 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,15 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2871,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index a326c8c..ec3bc62 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -644,6 +646,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1410,6 +1414,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..c31234e 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,9 @@
#include "testpmd.h"
+int testpmd_fd_copy; /* the copy of STDIN_FILENO */
+struct cmdline *testpmd_cl;
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +128,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -977,6 +983,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1070,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2018,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2533,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2655,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2797,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2867,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3099,7 +3134,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3621,6 +3656,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3719,13 +3758,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3756,12 +3788,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3786,10 +3822,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3917,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3932,6 +3966,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3967,6 +4003,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index a87ccb0..640a377 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+extern uint8_t f_quit;
+extern int testpmd_fd_copy;
+extern struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -423,6 +428,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..7e60e80 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,104 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``-a, --allow``
+
+ Add a device to the allow list. ``xxx`` means device used which should be the
+ same in primary process and secondary process.
+
+* ``--proc-type``
+
+ Specify a given process instance as the primary or secondary DPDK instance.
+ ``auto`` set here is OK.
+
+* ``-l CORELIST``
+
+ List of cores to run on. the corelist should be different in primary process and
+ secondary process.
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
@ 2021-03-30 2:17 ` Li, Xiaoyun
2021-03-30 6:36 ` Min Hu (Connor)
2021-03-30 3:11 ` Ajit Khaparde
2021-04-12 16:37 ` Ferruh Yigit
2 siblings, 1 reply; 80+ messages in thread
From: Li, Xiaoyun @ 2021-03-30 2:17 UTC (permalink / raw)
To: Min Hu (Connor), dev; +Cc: Yigit, Ferruh, ajit.khaparde
Hi
> -----Original Message-----
> From: Min Hu (Connor) <humin29@huawei.com>
> Sent: Tuesday, March 30, 2021 09:48
> To: dev@dpdk.org
> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com; Li,
> Xiaoyun <xiaoyun.li@intel.com>
> Subject: [PATCH v8] app/testpmd: support multi-process
>
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> v8:
> * Added warning info about queue numbers and process numbers.
>
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 12 +++-
> app/test-pmd/config.c | 14 +++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
> app/test-pmd/testpmd.h | 7 ++
> doc/guides/testpmd_app_ug/run_app.rst | 101
> +++++++++++++++++++++++++++
> 6 files changed, 226 insertions(+), 46 deletions(-)
> 2.7.4
Many commands shouldn't be allowed but now only have a brief guide in doc. This can be done in the future I think.
Overall, it looks good to me.
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 2:17 ` Li, Xiaoyun
@ 2021-03-30 6:36 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-30 6:36 UTC (permalink / raw)
To: Li, Xiaoyun, dev; +Cc: Yigit, Ferruh, ajit.khaparde
在 2021/3/30 10:17, Li, Xiaoyun 写道:
> Hi
>
>> -----Original Message-----
>> From: Min Hu (Connor) <humin29@huawei.com>
>> Sent: Tuesday, March 30, 2021 09:48
>> To: dev@dpdk.org
>> Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; ajit.khaparde@broadcom.com; Li,
>> Xiaoyun <xiaoyun.li@intel.com>
>> Subject: [PATCH v8] app/testpmd: support multi-process
>>
>> From: Lijun Ou <oulijun@huawei.com>
>>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> ---
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 12 +++-
>> app/test-pmd/config.c | 14 +++-
>> app/test-pmd/parameters.c | 11 +++
>> app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
>> app/test-pmd/testpmd.h | 7 ++
>> doc/guides/testpmd_app_ug/run_app.rst | 101
>> +++++++++++++++++++++++++++
>> 6 files changed, 226 insertions(+), 46 deletions(-)
>> 2.7.4
>
> Many commands shouldn't be allowed but now only have a brief guide in doc. This can be done in the future I think.
> Overall, it looks good to me.
>
Thanks xiaoyun.
You are right, Maybe there are more things to do about
supporting multiple process for testpmd. We will perfect
it continuously in future.
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
2021-03-30 2:17 ` Li, Xiaoyun
@ 2021-03-30 3:11 ` Ajit Khaparde
2021-03-30 6:41 ` Min Hu (Connor)
2021-04-12 16:37 ` Ferruh Yigit
2 siblings, 1 reply; 80+ messages in thread
From: Ajit Khaparde @ 2021-03-30 3:11 UTC (permalink / raw)
To: Min Hu (Connor); +Cc: dpdk-dev, Ferruh Yigit, Xiaoyun Li
[-- Attachment #1: Type: text/plain, Size: 2300 bytes --]
On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com> wrote:
>
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
Some minor nits below. Otherwise looks fine to me.
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> ---
> v8:
> * Added warning info about queue numbers and process numbers.
>
:::snip::::
> +* ``--rxq=N``
> +
> + Set the number of Rx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
of the number of processes.
> +
> +* ``--txq=N``
> +
> + Set the number of Tx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
of the number of processes.
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
:::: snip ::::
> +The number of rings should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. After RSS is
> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
> +Some traffic enters redundant queues and cannot be forwarded.
> +
> +Most dev ops is supported in primary and secondary process. While secondary process
Most dev ops are supported in the primary and secondary process. While....
> +is not permitted to allocate or release shared memory, so some ops are not supported
> +as follows:
> +``dev_configure``
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
:::: snip:::
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 3:11 ` Ajit Khaparde
@ 2021-03-30 6:41 ` Min Hu (Connor)
2021-03-30 10:19 ` Ferruh Yigit
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-30 6:41 UTC (permalink / raw)
To: Ajit Khaparde; +Cc: dpdk-dev, Ferruh Yigit, Xiaoyun Li
在 2021/3/30 11:11, Ajit Khaparde 写道:
> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com> wrote:
>>
>> From: Lijun Ou <oulijun@huawei.com>
>>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Some minor nits below. Otherwise looks fine to me.
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>
Thanks Ajit.
Hi, Ferruh,
Should I send v9 to fix the grammar bugs in doc which Ajit point
out or fix it in future?
>> ---
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
> :::snip::::
>
>> +* ``--rxq=N``
>> +
>> + Set the number of Rx queues per port to N. N is the sum of queues used by primary
>> + and secondary process. Primary process and secondary process should have separate
>> + queues, and each should occupy at least one queue. Where N should be the multiple
>> + of number of processes.
> of the number of processes.
>
>> +
>> +* ``--txq=N``
>> +
>> + Set the number of Tx queues per port to N. N is the sum of queues used by primary
>> + and secondary process. Primary process and secondary process should have separate
>> + queues, and each should occupy at least one queue. Where N should be the multiple
>> + of number of processes.
> of the number of processes.
>
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
> :::: snip ::::
>> +The number of rings should be a multiple of the number of processes. If not,
>> +redundant queues will exist after queues are allocated to processes. After RSS is
>> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
>> +Some traffic enters redundant queues and cannot be forwarded.
>> +
>> +Most dev ops is supported in primary and secondary process. While secondary process
> Most dev ops are supported in the primary and secondary process. While....
>
>> +is not permitted to allocate or release shared memory, so some ops are not supported
>> +as follows:
>> +``dev_configure``
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
> :::: snip:::
>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 6:41 ` Min Hu (Connor)
@ 2021-03-30 10:19 ` Ferruh Yigit
2021-03-30 10:43 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-03-30 10:19 UTC (permalink / raw)
To: Min Hu (Connor), Ajit Khaparde; +Cc: dpdk-dev, Xiaoyun Li
On 3/30/2021 7:41 AM, Min Hu (Connor) wrote:
>
>
> 在 2021/3/30 11:11, Ajit Khaparde 写道:
>> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com> wrote:
>>>
>>> From: Lijun Ou <oulijun@huawei.com>
>>>
>>> This patch adds multi-process support for testpmd.
>>> The test cmd example as follows:
>>> the primary cmd:
>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>
>>> the secondary cmd:
>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>
>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Some minor nits below. Otherwise looks fine to me.
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>
> Thanks Ajit.
>
> Hi, Ferruh,
> Should I send v9 to fix the grammar bugs in doc which Ajit point
> out or fix it in future?
>
Hi Connor, if they are only outstanding issues, I can fix them while merging.
>>> ---
>>> v8:
>>> * Added warning info about queue numbers and process numbers.
>>>
>> :::snip::::
>>
>>> +* ``--rxq=N``
>>> +
>>> + Set the number of Rx queues per port to N. N is the sum of queues used
>>> by primary
>>> + and secondary process. Primary process and secondary process should have
>>> separate
>>> + queues, and each should occupy at least one queue. Where N should be the
>>> multiple
>>> + of number of processes.
>> of the number of processes.
>>
>>> +
>>> +* ``--txq=N``
>>> +
>>> + Set the number of Tx queues per port to N. N is the sum of queues used
>>> by primary
>>> + and secondary process. Primary process and secondary process should have
>>> separate
>>> + queues, and each should occupy at least one queue. Where N should be the
>>> multiple
>>> + of number of processes.
>> of the number of processes.
>>
>>> +
>>> +* ``--num-procs=N``
>>> +
>>> + The number of processes which will be used.
>>> +
>> :::: snip ::::
>>> +The number of rings should be a multiple of the number of processes. If not,
>>> +redundant queues will exist after queues are allocated to processes. After
>>> RSS is
>>> +enabled, packet loss occurs when traffic is sent to all processes at the
>>> same time.
>>> +Some traffic enters redundant queues and cannot be forwarded.
>>> +
>>> +Most dev ops is supported in primary and secondary process. While secondary
>>> process
>> Most dev ops are supported in the primary and secondary process. While....
>>
>>> +is not permitted to allocate or release shared memory, so some ops are not
>>> supported
>>> +as follows:
>>> +``dev_configure``
>>> +``dev_start``
>>> +``dev_stop``
>>> +``rx_queue_setup``
>>> +``tx_queue_setup``
>>> +``rx_queue_release``
>>> +``tx_queue_release``
>> :::: snip:::
>>
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 10:19 ` Ferruh Yigit
@ 2021-03-30 10:43 ` Min Hu (Connor)
2021-04-08 10:32 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-03-30 10:43 UTC (permalink / raw)
To: Ferruh Yigit, Ajit Khaparde; +Cc: dpdk-dev, Xiaoyun Li
在 2021/3/30 18:19, Ferruh Yigit 写道:
> On 3/30/2021 7:41 AM, Min Hu (Connor) wrote:
>>
>>
>> 在 2021/3/30 11:11, Ajit Khaparde 写道:
>>> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com>
>>> wrote:
>>>>
>>>> From: Lijun Ou <oulijun@huawei.com>
>>>>
>>>> This patch adds multi-process support for testpmd.
>>>> The test cmd example as follows:
>>>> the primary cmd:
>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>
>>>> the secondary cmd:
>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>
>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>> Some minor nits below. Otherwise looks fine to me.
>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>
>> Thanks Ajit.
>>
>> Hi, Ferruh,
>> Should I send v9 to fix the grammar bugs in doc which Ajit point
>> out or fix it in future?
>>
>
> Hi Connor, if they are only outstanding issues, I can fix them while
> merging.
>
Thanks Ferruh.
>>>> ---
>>>> v8:
>>>> * Added warning info about queue numbers and process numbers.
>>>>
>>> :::snip::::
>>>
>>>> +* ``--rxq=N``
>>>> +
>>>> + Set the number of Rx queues per port to N. N is the sum of
>>>> queues used by primary
>>>> + and secondary process. Primary process and secondary process
>>>> should have separate
>>>> + queues, and each should occupy at least one queue. Where N
>>>> should be the multiple
>>>> + of number of processes.
>>> of the number of processes.
>>>
>>>> +
>>>> +* ``--txq=N``
>>>> +
>>>> + Set the number of Tx queues per port to N. N is the sum of
>>>> queues used by primary
>>>> + and secondary process. Primary process and secondary process
>>>> should have separate
>>>> + queues, and each should occupy at least one queue. Where N
>>>> should be the multiple
>>>> + of number of processes.
>>> of the number of processes.
>>>
>>>> +
>>>> +* ``--num-procs=N``
>>>> +
>>>> + The number of processes which will be used.
>>>> +
>>> :::: snip ::::
>>>> +The number of rings should be a multiple of the number of
>>>> processes. If not,
>>>> +redundant queues will exist after queues are allocated to
>>>> processes. After RSS is
>>>> +enabled, packet loss occurs when traffic is sent to all processes
>>>> at the same time.
>>>> +Some traffic enters redundant queues and cannot be forwarded.
>>>> +
>>>> +Most dev ops is supported in primary and secondary process. While
>>>> secondary process
>>> Most dev ops are supported in the primary and secondary process.
>>> While....
>>>
>>>> +is not permitted to allocate or release shared memory, so some ops
>>>> are not supported
>>>> +as follows:
>>>> +``dev_configure``
>>>> +``dev_start``
>>>> +``dev_stop``
>>>> +``rx_queue_setup``
>>>> +``tx_queue_setup``
>>>> +``rx_queue_release``
>>>> +``tx_queue_release``
>>> :::: snip:::
>>>
>
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 10:43 ` Min Hu (Connor)
@ 2021-04-08 10:32 ` Min Hu (Connor)
2021-04-08 13:27 ` Ferruh Yigit
0 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-08 10:32 UTC (permalink / raw)
To: Ferruh Yigit, Ajit Khaparde; +Cc: dpdk-dev, Xiaoyun Li
Hi, Ferry and all,
This patch has been acked:
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Are there any other comments about that patch?
在 2021/3/30 18:43, Min Hu (Connor) 写道:
>
>
> 在 2021/3/30 18:19, Ferruh Yigit 写道:
>> On 3/30/2021 7:41 AM, Min Hu (Connor) wrote:
>>>
>>>
>>> 在 2021/3/30 11:11, Ajit Khaparde 写道:
>>>> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com>
>>>> wrote:
>>>>>
>>>>> From: Lijun Ou <oulijun@huawei.com>
>>>>>
>>>>> This patch adds multi-process support for testpmd.
>>>>> The test cmd example as follows:
>>>>> the primary cmd:
>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>
>>>>> the secondary cmd:
>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>
>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>> Some minor nits below. Otherwise looks fine to me.
>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>
>>> Thanks Ajit.
>>>
>>> Hi, Ferruh,
>>> Should I send v9 to fix the grammar bugs in doc which Ajit point
>>> out or fix it in future?
>>>
>>
>> Hi Connor, if they are only outstanding issues, I can fix them while
>> merging.
>>
> Thanks Ferruh.
>>>>> ---
>>>>> v8:
>>>>> * Added warning info about queue numbers and process numbers.
>>>>>
>>>> :::snip::::
>>>>
>>>>> +* ``--rxq=N``
>>>>> +
>>>>> + Set the number of Rx queues per port to N. N is the sum of
>>>>> queues used by primary
>>>>> + and secondary process. Primary process and secondary process
>>>>> should have separate
>>>>> + queues, and each should occupy at least one queue. Where N
>>>>> should be the multiple
>>>>> + of number of processes.
>>>> of the number of processes.
>>>>
>>>>> +
>>>>> +* ``--txq=N``
>>>>> +
>>>>> + Set the number of Tx queues per port to N. N is the sum of
>>>>> queues used by primary
>>>>> + and secondary process. Primary process and secondary process
>>>>> should have separate
>>>>> + queues, and each should occupy at least one queue. Where N
>>>>> should be the multiple
>>>>> + of number of processes.
>>>> of the number of processes.
>>>>
>>>>> +
>>>>> +* ``--num-procs=N``
>>>>> +
>>>>> + The number of processes which will be used.
>>>>> +
>>>> :::: snip ::::
>>>>> +The number of rings should be a multiple of the number of
>>>>> processes. If not,
>>>>> +redundant queues will exist after queues are allocated to
>>>>> processes. After RSS is
>>>>> +enabled, packet loss occurs when traffic is sent to all processes
>>>>> at the same time.
>>>>> +Some traffic enters redundant queues and cannot be forwarded.
>>>>> +
>>>>> +Most dev ops is supported in primary and secondary process. While
>>>>> secondary process
>>>> Most dev ops are supported in the primary and secondary process.
>>>> While....
>>>>
>>>>> +is not permitted to allocate or release shared memory, so some ops
>>>>> are not supported
>>>>> +as follows:
>>>>> +``dev_configure``
>>>>> +``dev_start``
>>>>> +``dev_stop``
>>>>> +``rx_queue_setup``
>>>>> +``tx_queue_setup``
>>>>> +``rx_queue_release``
>>>>> +``tx_queue_release``
>>>> :::: snip:::
>>>>
>>
>> .
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-04-08 10:32 ` Min Hu (Connor)
@ 2021-04-08 13:27 ` Ferruh Yigit
2021-04-09 0:45 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-08 13:27 UTC (permalink / raw)
To: Min Hu (Connor), Ajit Khaparde; +Cc: dpdk-dev, Xiaoyun Li
On 4/8/2021 11:32 AM, Min Hu (Connor) wrote:
> Hi, Ferry and all,
> This patch has been acked:
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>
> Are there any other comments about that patch?
>
Hi Connor,
In today's release status meeting I asked if more parties can test this patch,
Lets wait a little more for it to be tested.
>
> 在 2021/3/30 18:43, Min Hu (Connor) 写道:
>>
>>
>> 在 2021/3/30 18:19, Ferruh Yigit 写道:
>>> On 3/30/2021 7:41 AM, Min Hu (Connor) wrote:
>>>>
>>>>
>>>> 在 2021/3/30 11:11, Ajit Khaparde 写道:
>>>>> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor) <humin29@huawei.com> wrote:
>>>>>>
>>>>>> From: Lijun Ou <oulijun@huawei.com>
>>>>>>
>>>>>> This patch adds multi-process support for testpmd.
>>>>>> The test cmd example as follows:
>>>>>> the primary cmd:
>>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>>
>>>>>> the secondary cmd:
>>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>>
>>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>> Some minor nits below. Otherwise looks fine to me.
>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>>
>>>> Thanks Ajit.
>>>>
>>>> Hi, Ferruh,
>>>> Should I send v9 to fix the grammar bugs in doc which Ajit point
>>>> out or fix it in future?
>>>>
>>>
>>> Hi Connor, if they are only outstanding issues, I can fix them while merging.
>>>
>> Thanks Ferruh.
>>>>>> ---
>>>>>> v8:
>>>>>> * Added warning info about queue numbers and process numbers.
>>>>>>
>>>>> :::snip::::
>>>>>
>>>>>> +* ``--rxq=N``
>>>>>> +
>>>>>> + Set the number of Rx queues per port to N. N is the sum of queues
>>>>>> used by primary
>>>>>> + and secondary process. Primary process and secondary process should
>>>>>> have separate
>>>>>> + queues, and each should occupy at least one queue. Where N should be
>>>>>> the multiple
>>>>>> + of number of processes.
>>>>> of the number of processes.
>>>>>
>>>>>> +
>>>>>> +* ``--txq=N``
>>>>>> +
>>>>>> + Set the number of Tx queues per port to N. N is the sum of queues
>>>>>> used by primary
>>>>>> + and secondary process. Primary process and secondary process should
>>>>>> have separate
>>>>>> + queues, and each should occupy at least one queue. Where N should be
>>>>>> the multiple
>>>>>> + of number of processes.
>>>>> of the number of processes.
>>>>>
>>>>>> +
>>>>>> +* ``--num-procs=N``
>>>>>> +
>>>>>> + The number of processes which will be used.
>>>>>> +
>>>>> :::: snip ::::
>>>>>> +The number of rings should be a multiple of the number of processes. If not,
>>>>>> +redundant queues will exist after queues are allocated to processes.
>>>>>> After RSS is
>>>>>> +enabled, packet loss occurs when traffic is sent to all processes at the
>>>>>> same time.
>>>>>> +Some traffic enters redundant queues and cannot be forwarded.
>>>>>> +
>>>>>> +Most dev ops is supported in primary and secondary process. While
>>>>>> secondary process
>>>>> Most dev ops are supported in the primary and secondary process. While....
>>>>>
>>>>>> +is not permitted to allocate or release shared memory, so some ops are
>>>>>> not supported
>>>>>> +as follows:
>>>>>> +``dev_configure``
>>>>>> +``dev_start``
>>>>>> +``dev_stop``
>>>>>> +``rx_queue_setup``
>>>>>> +``tx_queue_setup``
>>>>>> +``rx_queue_release``
>>>>>> +``tx_queue_release``
>>>>> :::: snip:::
>>>>>
>>>
>>> .
>> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-04-08 13:27 ` Ferruh Yigit
@ 2021-04-09 0:45 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-09 0:45 UTC (permalink / raw)
To: Ferruh Yigit, Ajit Khaparde; +Cc: dpdk-dev, Xiaoyun Li
在 2021/4/8 21:27, Ferruh Yigit 写道:
> On 4/8/2021 11:32 AM, Min Hu (Connor) wrote:
>> Hi, Ferry and all,
>> This patch has been acked:
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>
>> Are there any other comments about that patch?
>>
>
> Hi Connor,
>
> In today's release status meeting I asked if more parties can test this
> patch,
> Lets wait a little more for it to be tested.
>
OK, thanks Ferruh.
>>
>> 在 2021/3/30 18:43, Min Hu (Connor) 写道:
>>>
>>>
>>> 在 2021/3/30 18:19, Ferruh Yigit 写道:
>>>> On 3/30/2021 7:41 AM, Min Hu (Connor) wrote:
>>>>>
>>>>>
>>>>> 在 2021/3/30 11:11, Ajit Khaparde 写道:
>>>>>> On Mon, Mar 29, 2021 at 6:48 PM Min Hu (Connor)
>>>>>> <humin29@huawei.com> wrote:
>>>>>>>
>>>>>>> From: Lijun Ou <oulijun@huawei.com>
>>>>>>>
>>>>>>> This patch adds multi-process support for testpmd.
>>>>>>> The test cmd example as follows:
>>>>>>> the primary cmd:
>>>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>>>>>
>>>>>>> the secondary cmd:
>>>>>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>>>>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>>>>>
>>>>>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>>>>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>>>>> Some minor nits below. Otherwise looks fine to me.
>>>>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>>>>>
>>>>> Thanks Ajit.
>>>>>
>>>>> Hi, Ferruh,
>>>>> Should I send v9 to fix the grammar bugs in doc which Ajit point
>>>>> out or fix it in future?
>>>>>
>>>>
>>>> Hi Connor, if they are only outstanding issues, I can fix them while
>>>> merging.
>>>>
>>> Thanks Ferruh.
>>>>>>> ---
>>>>>>> v8:
>>>>>>> * Added warning info about queue numbers and process numbers.
>>>>>>>
>>>>>> :::snip::::
>>>>>>
>>>>>>> +* ``--rxq=N``
>>>>>>> +
>>>>>>> + Set the number of Rx queues per port to N. N is the sum of
>>>>>>> queues used by primary
>>>>>>> + and secondary process. Primary process and secondary process
>>>>>>> should have separate
>>>>>>> + queues, and each should occupy at least one queue. Where N
>>>>>>> should be the multiple
>>>>>>> + of number of processes.
>>>>>> of the number of processes.
>>>>>>
>>>>>>> +
>>>>>>> +* ``--txq=N``
>>>>>>> +
>>>>>>> + Set the number of Tx queues per port to N. N is the sum of
>>>>>>> queues used by primary
>>>>>>> + and secondary process. Primary process and secondary process
>>>>>>> should have separate
>>>>>>> + queues, and each should occupy at least one queue. Where N
>>>>>>> should be the multiple
>>>>>>> + of number of processes.
>>>>>> of the number of processes.
>>>>>>
>>>>>>> +
>>>>>>> +* ``--num-procs=N``
>>>>>>> +
>>>>>>> + The number of processes which will be used.
>>>>>>> +
>>>>>> :::: snip ::::
>>>>>>> +The number of rings should be a multiple of the number of
>>>>>>> processes. If not,
>>>>>>> +redundant queues will exist after queues are allocated to
>>>>>>> processes. After RSS is
>>>>>>> +enabled, packet loss occurs when traffic is sent to all
>>>>>>> processes at the same time.
>>>>>>> +Some traffic enters redundant queues and cannot be forwarded.
>>>>>>> +
>>>>>>> +Most dev ops is supported in primary and secondary process.
>>>>>>> While secondary process
>>>>>> Most dev ops are supported in the primary and secondary process.
>>>>>> While....
>>>>>>
>>>>>>> +is not permitted to allocate or release shared memory, so some
>>>>>>> ops are not supported
>>>>>>> +as follows:
>>>>>>> +``dev_configure``
>>>>>>> +``dev_start``
>>>>>>> +``dev_stop``
>>>>>>> +``rx_queue_setup``
>>>>>>> +``tx_queue_setup``
>>>>>>> +``rx_queue_release``
>>>>>>> +``tx_queue_release``
>>>>>> :::: snip:::
>>>>>>
>>>>
>>>> .
>>> .
>
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
2021-03-30 2:17 ` Li, Xiaoyun
2021-03-30 3:11 ` Ajit Khaparde
@ 2021-04-12 16:37 ` Ferruh Yigit
2021-04-15 7:54 ` Ferruh Yigit
2 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-12 16:37 UTC (permalink / raw)
To: Min Hu (Connor), dev; +Cc: ajit.khaparde, xiaoyun.li
On 3/30/2021 2:48 AM, Min Hu (Connor) wrote:
> From: Lijun Ou <oulijun@huawei.com>
>
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
Hi Connor,
Please find a few minor comments below, since they are minor issues please feel
free to keep the existing acks in next version.
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> ---
> v8:
> * Added warning info about queue numbers and process numbers.
>
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 12 +++-
> app/test-pmd/config.c | 14 +++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++------------
> app/test-pmd/testpmd.h | 7 ++
> doc/guides/testpmd_app_ug/run_app.rst | 101 +++++++++++++++++++++++++++
Can you please add a release notes update too?
> 6 files changed, 226 insertions(+), 46 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index f44116b..38c97e3 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -71,8 +71,6 @@
> #include "cmdline_tm.h"
> #include "bpf_cmd.h"
>
> -static struct cmdline *testpmd_cl;
> -
> static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
>
> /* *** Help command with introduction. *** */
> @@ -5351,6 +5349,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
> __rte_unused void *data)
> {
> struct cmd_set_flush_rx *res = parsed_result;
> +
> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> + printf("multi-process doesn't support to flush rx queues.\n");
> + return;
> + }
> +
> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
> }
>
> @@ -17229,6 +17233,10 @@ prompt(void)
> printf("Cannot set exit function for cmdline\n");
>
> cmdline_interact(testpmd_cl);
> + if (unlikely(f_quit == 1)) {
> + dup2(testpmd_fd_copy, testpmd_cl->s_in);
> + close(testpmd_fd_copy);
> + }
Why this is needed? Can you put some comment on the code to explain why this is
needed?
> if (ret != 0)
> cmdline_stdin_exit(testpmd_cl);
> }
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index ef0b978..ec7d030 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
> queueid_t rxq;
> queueid_t nb_q;
> streamid_t sm_id;
> + int start;
> + int end;
>
> nb_q = nb_rxq;
> if (nb_q > nb_txq)
> @@ -2844,7 +2846,15 @@ rss_fwd_config_setup(void)
> init_fwd_streams();
>
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> +
> + if (proc_id > 0 && nb_q % num_procs)
> + printf("Warning! queue numbers should be multiple of "
> + "processes, or packet loss will happen.\n");
> +
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -2861,6 +2871,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index a326c8c..ec3bc62 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -45,6 +45,8 @@
> #include <rte_flow.h>
>
> #include "testpmd.h"
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
>
Macros just after include is not good, can you please move these just after
'launch_args_parse()' to keep locality.
> static void
> usage(char* progname)
> @@ -644,6 +646,8 @@ launch_args_parse(int argc, char** argv)
> { "rx-mq-mode", 1, 0, 0 },
> { "record-core-cycles", 0, 0, 0 },
> { "record-burst-stats", 0, 0, 0 },
> + { PARAM_NUM_PROCS, 1, 0, 0 },
> + { PARAM_PROC_ID, 1, 0, 0 },
> { 0, 0, 0, 0 },
> };
>
> @@ -1410,6 +1414,13 @@ launch_args_parse(int argc, char** argv)
> record_core_cycles = 1;
> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
> record_burst_stats = 1;
> +
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_NUM_PROCS, 8) == 0)
9?
"num-procs"
123456789
> + num_procs = atoi(optarg);
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_PROC_ID, 7) == 0)
> + proc_id = atoi(optarg);
> break;
> case 'h':
> usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 96d2e0f..c31234e 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -63,6 +63,9 @@
>
> #include "testpmd.h"
>
> +int testpmd_fd_copy; /* the copy of STDIN_FILENO */
> +struct cmdline *testpmd_cl;
> +
Again the global variables just after include is not good place.
What do you think to keep 'testpmd_cl' in the 'cmdline.c', since the object
allocated there and the variable seems match more to that file. 'extern' already
added to header, so it can be used here.
'testpmd_fd_copy' can be moved at the bottom of the block that has global
variables, and please add more comment on what this variable is for.
> #ifndef MAP_HUGETLB
> /* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
> #define HUGE_FLAG (0x40000)
> @@ -125,6 +128,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
> */
> uint8_t rxring_numa[RTE_MAX_ETHPORTS];
>
> +int proc_id;
> +unsigned int num_procs = 1;
> +
Similar comments for these global variables.
The testpmd is already complex, it is good to have multi process support for the
testpmd but the concern is additional complexity it bring, it may lead very
tangled senarious. Please add much more comment on the multi process related
additions.
There is a block at the beggining of the this file, that has all global
variables with enough comments on them, can we follow same for these new global
variables.
Group all multiprocess related variables together, appent to the global
variables block, and comment all the variables in detail?
> /*
> * Store specified sockets on which TX ring to be used by ports
> * is allocated.
> @@ -977,6 +983,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
> mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
> mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
>
> + if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
> + rte_mp = rte_mempool_lookup(pool_name);
> + goto err;
> + }
> +
> TESTPMD_LOG(INFO,
> "create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
> pool_name, nb_mbuf, mbuf_seg_size, socket_id);
> @@ -1059,9 +1070,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
>
> err:
> if (rte_mp == NULL) {
> - rte_exit(EXIT_FAILURE,
> - "Creation of mbuf pool for socket %u failed: %s\n",
> - socket_id, rte_strerror(rte_errno));
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_exit(EXIT_FAILURE,
> + "Creation of mbuf pool for socket %u failed: %s\n",
> + socket_id, rte_strerror(rte_errno));
> + else
> + rte_exit(EXIT_FAILURE,
> + "Get mbuf pool for socket %u failed: %s\n",
> + socket_id, rte_strerror(rte_errno));
> } else if (verbose_level > 0) {
> rte_mempool_dump(stdout, rte_mp);
> }
> @@ -2002,6 +2018,12 @@ flush_fwd_rx_queues(void)
> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
> uint64_t timer_period;
>
> + if (num_procs > 1) {
> + printf("multi-process not support for flushing fwd rx "
> + "queues, skip the below lines and return.\n");
> + return;
> + }
> +
> /* convert to number of cycles */
> timer_period = rte_get_timer_hz(); /* 1 second timeout */
>
> @@ -2511,21 +2533,28 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_configure(pi,
> + nb_rxq + nb_hairpinq,
> nb_txq + nb_hairpinq,
> &(port->dev_conf));
> - if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> - printf("Fail to configure port %d\n", pi);
> - /* try to reconfigure port next time */
> - port->need_reconfig = 1;
> - return -1;
> + if (diag != 0) {
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set "
> + "back to stopped\n", pi);
> + printf("Fail to configure port %d\n",
> + pi);
> + /* try to reconfigure port next time */
> + port->need_reconfig = 1;
> + return -1;
> + }
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 &&
> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2626,17 +2655,20 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> - if (diag < 0) {
> - printf("Fail to start port %d: %s\n", pi,
> - rte_strerror(-diag));
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_start(pi);
> + if (diag < 0) {
> + printf("Fail to start port %d: %s\n", pi,
> + rte_strerror(-diag));
>
> - /* Fail to setup rx queue, return */
> - if (rte_atomic16_cmpset(&(port->port_status),
> + /* Fail to setup rx queue, return */
> + if (rte_atomic16_cmpset(&(port->port_status),
> RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> - "stopped\n", pi);
> - continue;
> + printf("Port %d can not be set back to "
> + "stopped\n",
> + pi);
> + continue;
> + }
> }
>
> if (rte_atomic16_cmpset(&(port->port_status),
> @@ -2765,7 +2797,8 @@ stop_port(portid_t pid)
> if (port->flow_list)
> port_flow_flush(pi);
>
> - if (rte_eth_dev_stop(pi) != 0)
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
> + rte_eth_dev_stop(pi) != 0)
> RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
> pi);
>
> @@ -2834,8 +2867,10 @@ close_port(portid_t pid)
> continue;
> }
>
> - port_flow_flush(pi);
> - rte_eth_dev_close(pi);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + port_flow_flush(pi);
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> + rte_eth_dev_close(pi);
Can combine two calls to prevent redundant check.
> }
>
> remove_invalid_ports();
> @@ -3099,7 +3134,7 @@ pmd_test_exit(void)
> }
> }
> for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
> - if (mempools[i])
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
> rte_mempool_free(mempools[i]);
> }
>
> @@ -3621,6 +3656,10 @@ init_port_dcb_config(portid_t pid,
> int retval;
> uint16_t i;
>
> + if (num_procs > 1) {
> + printf("The multi-process feature doesn't support dcb.\n");
> + return -ENOTSUP;
> + }
> rte_port = &ports[pid];
>
> memset(&port_conf, 0, sizeof(struct rte_eth_conf));
> @@ -3719,13 +3758,6 @@ init_port(void)
> }
>
> static void
> -force_quit(void)
> -{
> - pmd_test_exit();
> - prompt_exit();
> -}
> -
> -static void
> print_stats(void)
> {
> uint8_t i;
> @@ -3756,12 +3788,16 @@ signal_handler(int signum)
> if (latencystats_enabled != 0)
> rte_latencystats_uninit();
> #endif
> - force_quit();
> /* Set flag to indicate the force termination. */
> f_quit = 1;
> - /* exit with the expected status */
> - signal(signum, SIG_DFL);
> - kill(getpid(), signum);
> + if (interactive == 1) {
> + dup2(testpmd_cl->s_in, testpmd_fd_copy);
> + close(testpmd_cl->s_in);
> + } else {
> + dup2(0, testpmd_fd_copy);
> + close(0);
> + }
> +
Similarly can you please comment the code why this is needed.
> }
> }
>
> @@ -3786,10 +3822,6 @@ main(int argc, char** argv)
> rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
> rte_strerror(rte_errno));
>
> - if (rte_eal_process_type() == RTE_PROC_SECONDARY)
> - rte_exit(EXIT_FAILURE,
> - "Secondary process type not supported.\n");
> -
> ret = register_eth_event_callback();
> if (ret != 0)
> rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
> @@ -3885,8 +3917,10 @@ main(int argc, char** argv)
> }
> }
>
> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
> + pmd_test_exit();
> rte_exit(EXIT_FAILURE, "Start ports failed\n");
> + }
>
> /* set all ports to promiscuous mode by default */
> RTE_ETH_FOREACH_DEV(port_id) {
> @@ -3932,6 +3966,8 @@ main(int argc, char** argv)
> }
> prompt();
> pmd_test_exit();
> + if (unlikely(f_quit == 1))
> + prompt_exit();
Why additional prompt_exit() is required?
> } else
> #endif
> {
> @@ -3967,6 +4003,11 @@ main(int argc, char** argv)
> printf("Press enter to exit\n");
> rc = read(0, &c, 1);
> pmd_test_exit();
> + if (unlikely(f_quit == 1)) {
> + dup2(testpmd_fd_copy, 0);
> + close(testpmd_fd_copy);
> + prompt_exit();
> + }
By definition 'f_quit', force quit, is added to support exiting in container
environment, since they can't exit with SIGINT/SIGTERM, so main thing with this
global variable should be breaking the loop in main function.
I wonder why this variable used extensively in this patch, is it used in
different purpose here? will quit/Ctrl-C behaviour differ in multi process?
> if (rc < 0)
> return 1;
> }
> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
> index a87ccb0..640a377 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
> @@ -13,6 +13,7 @@
> #include <rte_gso.h>
> #include <cmdline.h>
> #include <sys/queue.h>
> +#include "cmdline.h"
>
> #define RTE_PORT_ALL (~(portid_t)0x0)
>
> @@ -24,6 +25,10 @@
> #define RTE_PORT_CLOSED (uint16_t)2
> #define RTE_PORT_HANDLING (uint16_t)3
>
> +extern uint8_t f_quit;
> +extern int testpmd_fd_copy;
> +extern struct cmdline *testpmd_cl;
> +
Again can you please move them in the file, there is already a block in below
for extern variables, please place there with sufficient comments.
> /*
> * It is used to allocate the memory for hash key.
> * The hash key size is NIC dependent.
> @@ -423,6 +428,8 @@ extern uint64_t noisy_lkup_mem_sz;
> extern uint64_t noisy_lkup_num_writes;
> extern uint64_t noisy_lkup_num_reads;
> extern uint64_t noisy_lkup_num_reads_writes;
> +extern int proc_id;
> +extern unsigned int num_procs;
>
Can you pleaes group multi process related changes together?
> extern uint8_t dcb_config;
> extern uint8_t dcb_test;
> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
> index ec1dc7d..7e60e80 100644
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -551,3 +551,104 @@ The command line options are:
> bit 1 - two hairpin ports paired
> bit 0 - two hairpin ports loop
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +
> +Testpmd Support Multi Process Command-line Options
> +--------------------------------------------------
> +
> +The following are the command-line options for the testpmd applications(support
> +multi process).They must be separated from the EAL options, shown in the previous
> +section, with a ``--`` separator:
> +
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=1
> +
> +The command line options are:
> +
> +* ``-a, --allow``
> +
> + Add a device to the allow list. ``xxx`` means device used which should be the
> + same in primary process and secondary process.
> +
> +* ``--proc-type``
> +
> + Specify a given process instance as the primary or secondary DPDK instance.
> + ``auto`` set here is OK.
> +
> +* ``-l CORELIST``
> +
> + List of cores to run on. the corelist should be different in primary process and
> + secondary process.
> +
+1 for documentation, but above are eal parameters, what about not repeting them
here but referencing eal documentation,
'doc/guides/linux_gsg/eal_args.include.rst', for it?
> +* ``--rxq=N``
> +
> + Set the number of Rx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--txq=N``
> +
> + Set the number of Tx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary
> + process and secondary process.
> +
> +Calculation rule for queue:
> +All queues are allocated to different processes based on proc_num and proc_id.
> +Calculation rule for the Testpmd to allocate queues to each process:
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
> +For example, if supports 4 txq and rxq
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +The number of rings should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. After RSS is
> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
> +Some traffic enters redundant queues and cannot be forwarded.
> +
> +Most dev ops is supported in primary and secondary process. While secondary process
> +is not permitted to allocate or release shared memory, so some ops are not supported
> +as follows:
> +``dev_configure``
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
> +
> +So, any command from testpmd which calls those APIs will not be supported in secondary
> +process, like:
> +``port config all rxq|txq|rxd|txd <value>``
> +``port config <port_id> rx_offload xxx on/off ``
> +``port config <port_id> tx_offload xxx on/off``
> +etc.
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
> +stats supported, stats will not change when one quit and start, As they share the same
> +buffer to store the stats. Flow rules are maintained in process level: primary and secondary
> +has its own flow list(but one flow list in HW). The two can see all the queues, so setting
> +the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
> +from others is not permitted.
> +
> +RSS supported, Primary process and secondary process has separate queues to use, RSS
> +will work in their own queues whether primary and secondary process.
>
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-04-12 16:37 ` Ferruh Yigit
@ 2021-04-15 7:54 ` Ferruh Yigit
2021-04-16 2:20 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-15 7:54 UTC (permalink / raw)
To: Min Hu (Connor), dev
Cc: ajit.khaparde, xiaoyun.li, Chen, Zhaoyan, Yu, PingX,
Aaron Conole, Honnappa Nagarahalli
On 4/12/2021 5:37 PM, Ferruh Yigit wrote:
> On 3/30/2021 2:48 AM, Min Hu (Connor) wrote:
>> From: Lijun Ou <oulijun@huawei.com>
>>
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>
> Hi Connor,
>
> Please find a few minor comments below, since they are minor issues please feel
> free to keep the existing acks in next version.
>
Reminder of this patch, waiting for minor updates, it would be good to have it
before -rc1, so that the change can be tested thoroughly.
Meanwhile I have concern/question on long term testing of the feature. It is
possible to break the multi-process support unintentionally and not recognize
it. How can we continually verify the feature?
Are you using DTS?
Does it make sense to add a testcase into dts to test testpmd multi-process
support so that it is verified on each release?
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v8] app/testpmd: support multi-process
2021-04-15 7:54 ` Ferruh Yigit
@ 2021-04-16 2:20 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-16 2:20 UTC (permalink / raw)
To: Ferruh Yigit, dev
Cc: ajit.khaparde, xiaoyun.li, Chen, Zhaoyan, Yu, PingX,
Aaron Conole, Honnappa Nagarahalli
在 2021/4/15 15:54, Ferruh Yigit 写道:
> On 4/12/2021 5:37 PM, Ferruh Yigit wrote:
>> On 3/30/2021 2:48 AM, Min Hu (Connor) wrote:
>>> From: Lijun Ou <oulijun@huawei.com>
>>>
>>> This patch adds multi-process support for testpmd.
>>> The test cmd example as follows:
>>> the primary cmd:
>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>
>>> the secondary cmd:
>>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>
>>
>> Hi Connor,
>>
>> Please find a few minor comments below, since they are minor issues
>> please feel free to keep the existing acks in next version.
>>
>
> Reminder of this patch, waiting for minor updates, it would be good to
> have it before -rc1, so that the change can be tested thoroughly.
> Hi, Ferruh,
v9 has been sent, please check it out, thanks.
> Meanwhile I have concern/question on long term testing of the feature.
> It is possible to break the multi-process support unintentionally and
> not recognize it. How can we continually verify the feature?
>
> Are you using DTS?
> Does it make sense to add a testcase into dts to test testpmd
> multi-process support so that it is verified on each release?
> .
Well, we have our own DTS system which contains testcase of multiple
process in testpmd:
1. When we upstream patches to DPDK，the patch should pass all the testcase.
2. Before DPDK version release(in RC1) in every three months, we rebase
the new DPDK version, and check all the testcase in our DTS system. If
some testcase fails, we will fix it and upstream the patch to DPDK, of
course, including multi-process support in testpmd.
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v9] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (7 preceding siblings ...)
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
@ 2021-04-16 1:52 ` Min Hu (Connor)
2021-04-16 16:30 ` Ferruh Yigit
2021-04-17 6:12 ` [dpdk-dev] [PATCH v10] " Min Hu (Connor)
` (3 subsequent siblings)
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-16 1:52 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
---
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 14 ++++-
app/test-pmd/parameters.c | 12 ++++
app/test-pmd/testpmd.c | 108 +++++++++++++++++++++++----------
app/test-pmd/testpmd.h | 3 +
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
7 files changed, 196 insertions(+), 34 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 5bf1497..e465824 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 40b2b29..c982c87 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..d86cc21 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..01d0d82 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process.
+ */
+int proc_id;
+
+/*
+ * Number of processes.
+ */
+unsigned int num_procs = 1;
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +987,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (rte_eal_process_type() != RTE_PROC_PRIMARY) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1074,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2022,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2537,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,17 +2659,20 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
- if (diag < 0) {
- printf("Fail to start port %d: %s\n", pi,
- rte_strerror(-diag));
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d: %s\n", pi,
+ rte_strerror(-diag));
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
- continue;
+ printf("Port %d can not be set back to "
+ "stopped\n",
+ pi);
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2765,7 +2801,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2871,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3099,7 +3138,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3430,7 +3469,8 @@ update_jumbo_frame_offload(portid_t portid)
/* If JUMBO_FRAME is set MTU conversion done by ethdev layer,
* if unset do it here
*/
- if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
+ if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
ret = rte_eth_dev_set_mtu(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
@@ -3621,6 +3661,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3786,10 +3830,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3885,8 +3925,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 36d8535..e04016c 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -630,6 +630,9 @@ extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
extern enum rte_eth_rx_mq_mode rx_mq_mode;
+extern int proc_id;
+extern unsigned int num_procs;
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 3bd7757..df01d4f 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -185,6 +185,7 @@ New Features
``show port (port_id) rxq (queue_id) desc used count``
* Added command to dump internal representation information of single flow.
``flow dump (port_id) rule (rule_id)``
+ * Added support multi-process for testpmd.
Removed Items
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..d714951 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,89 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v9] app/testpmd: support multi-process
2021-04-16 1:52 ` [dpdk-dev] [PATCH v9] " Min Hu (Connor)
@ 2021-04-16 16:30 ` Ferruh Yigit
2021-04-17 6:12 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-16 16:30 UTC (permalink / raw)
To: Min Hu (Connor), dev, Xiaoyun Li
Cc: Anatoly Burakov, Andrew Rybchenko, Thomas Monjalon
On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
Hi Connor,
Thanks for the update. +Anatoly as multi-process maintainer, and ethdev maintainers.
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> ---
> v9:
> * Updated release notes and rst doc.
> * Deleted deprecated codes.
> * move macro and variable.
>
> v8:
> * Added warning info about queue numbers and process numbers.
>
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 6 ++
> app/test-pmd/config.c | 14 ++++-
> app/test-pmd/parameters.c | 12 ++++
> app/test-pmd/testpmd.c | 108 +++++++++++++++++++++++----------
> app/test-pmd/testpmd.h | 3 +
> doc/guides/rel_notes/release_21_05.rst | 1 +
> doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
> 7 files changed, 196 insertions(+), 34 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index 5bf1497..e465824 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
> __rte_unused void *data)
> {
> struct cmd_set_flush_rx *res = parsed_result;
> +
> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> + printf("multi-process doesn't support to flush rx queues.\n");
> + return;
> + }
> +
> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
> }
>
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index 40b2b29..c982c87 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
> queueid_t rxq;
> queueid_t nb_q;
> streamid_t sm_id;
> + int start;
> + int end;
>
> nb_q = nb_rxq;
> if (nb_q > nb_txq)
> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
> init_fwd_streams();
>
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> +
> + if (proc_id > 0 && nb_q % num_procs)
> + printf("Warning! queue numbers should be multiple of "
> + "processes, or packet loss will happen.\n");
> +
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
Can you put some comment above, on what/why is done, something similar to you
already put into the documentation?
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index f3954c1..d86cc21 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -508,6 +508,9 @@ parse_link_speed(int n)
> void
> launch_args_parse(int argc, char** argv)
> {
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
> +
> int n, opt;
> char **argvopt;
> int opt_idx;
> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
> { "rx-mq-mode", 1, 0, 0 },
> { "record-core-cycles", 0, 0, 0 },
> { "record-burst-stats", 0, 0, 0 },
> + { PARAM_NUM_PROCS, 1, 0, 0 },
> + { PARAM_PROC_ID, 1, 0, 0 },
> { 0, 0, 0, 0 },
> };
>
> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
> record_core_cycles = 1;
> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
> record_burst_stats = 1;
> +
To be consistent for rest, this empty line can be removed.
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_NUM_PROCS, 9) == 0)
> + num_procs = atoi(optarg);
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_PROC_ID, 7) == 0)
> + proc_id = atoi(optarg);
> break;
> case 'h':
> usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index 96d2e0f..01d0d82 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
> */
> uint32_t eth_link_speed;
>
> +/*
> + * Id of the current process.
> + */
Can you please add more info, like this being related to the primary/secondary
process support and used to configure the queues to be polled.
> +int proc_id;
> +
> +/*
> + * Number of processes.
> + */
> +unsigned int num_procs = 1;
> +
Ditto.
<...>
> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_configure(pi,
> + nb_rxq + nb_hairpinq,
> nb_txq + nb_hairpinq,
> &(port->dev_conf));
> - if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> - printf("Fail to configure port %d\n", pi);
> - /* try to reconfigure port next time */
> - port->need_reconfig = 1;
> - return -1;
> + if (diag != 0) {
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set "
> + "back to stopped\n", pi);
> + printf("Fail to configure port %d\n",
> + pi);
> + /* try to reconfigure port next time */
> + port->need_reconfig = 1;
> + return -1;
> + }
Just an idea,
I am a little worried about the complexity this new process type checks are
added to the multiple locations. What do you think hiding process type checks
behind new functions, like:
eth_dev_configure_mp(...) {
if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
return rte_eth_dev_configure(..)
}
return 0;
}
and replace it with 'rte_eth_dev_configure()':
-diag = rte_eth_dev_configure(...)
+diag = eth_dev_configure_mp(...)
Do you think does this help reducing the complexity added by the multi-process
support?
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 &&
> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
According our coding convention, we don't allign with paranthesis but put double
tabs.
And what about creating an inline function for primary process check, like
is_proc_primary(), that may help.
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> - if (diag < 0) {
> - printf("Fail to start port %d: %s\n", pi,
> - rte_strerror(-diag));
> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> + diag = rte_eth_dev_start(pi);
> + if (diag < 0) {
> + printf("Fail to start port %d: %s\n", pi,
> + rte_strerror(-diag));
>
> - /* Fail to setup rx queue, return */
> - if (rte_atomic16_cmpset(&(port->port_status),
> + /* Fail to setup rx queue, return */
> + if (rte_atomic16_cmpset(&(port->port_status),
> RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
Syntax is wrong here after change, but if you go with above idea to hide process
type check within functions, you won't need to change these lines at all.
<...>
> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
> }
> }
>
> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
> + pmd_test_exit();
Why need to add 'pmd_test_exit()' for the multi-process support, if there is a
special case, please add comment for it.
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v9] app/testpmd: support multi-process
2021-04-16 16:30 ` Ferruh Yigit
@ 2021-04-17 6:12 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-17 6:12 UTC (permalink / raw)
To: Ferruh Yigit, dev, Xiaoyun Li
Cc: Anatoly Burakov, Andrew Rybchenko, Thomas Monjalon
Hi, Ferrruh,
All is fixed in v10, please check it out, thanks.
在 2021/4/17 0:30, Ferruh Yigit 写道:
> On 4/16/2021 2:52 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>
> Hi Connor,
>
> Thanks for the update. +Anatoly as multi-process maintainer, and ethdev
> maintainers.
>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> ---
>> v9:
>> * Updated release notes and rst doc.
>> * Deleted deprecated codes.
>> * move macro and variable.
>>
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 6 ++
>> app/test-pmd/config.c | 14 ++++-
>> app/test-pmd/parameters.c | 12 ++++
>> app/test-pmd/testpmd.c | 108
>> +++++++++++++++++++++++----------
>> app/test-pmd/testpmd.h | 3 +
>> doc/guides/rel_notes/release_21_05.rst | 1 +
>> doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++++++
>> 7 files changed, 196 insertions(+), 34 deletions(-)
>>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
>> index 5bf1497..e465824 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
>> __rte_unused void *data)
>> {
>> struct cmd_set_flush_rx *res = parsed_result;
>> +
>> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
>> + printf("multi-process doesn't support to flush rx queues.\n");
>> + return;
>> + }
>> +
>> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
>> }
>> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
>> index 40b2b29..c982c87 100644
>> --- a/app/test-pmd/config.c
>> +++ b/app/test-pmd/config.c
>> @@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
>> queueid_t rxq;
>> queueid_t nb_q;
>> streamid_t sm_id;
>> + int start;
>> + int end;
>> nb_q = nb_rxq;
>> if (nb_q > nb_txq)
>> @@ -2877,7 +2879,15 @@ rss_fwd_config_setup(void)
>> init_fwd_streams();
>> setup_fwd_config_of_each_lcore(&cur_fwd_config);
>> - rxp = 0; rxq = 0;
>> +
>> + if (proc_id > 0 && nb_q % num_procs)
>> + printf("Warning! queue numbers should be multiple of "
>> + "processes, or packet loss will happen.\n");
>> +
>> + start = proc_id * nb_q / num_procs;
>> + end = start + nb_q / num_procs;
>> + rxp = 0;
>> + rxq = start;
>
> Can you put some comment above, on what/why is done, something similar
> to you already put into the documentation?
>
>> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>> struct fwd_stream *fs;
>> @@ -2894,6 +2904,8 @@ rss_fwd_config_setup(void)
>> continue;
>> rxp = 0;
>> rxq++;
>> + if (rxq >= end)
>> + rxq = start;
>> }
>> }
>> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
>> index f3954c1..d86cc21 100644
>> --- a/app/test-pmd/parameters.c
>> +++ b/app/test-pmd/parameters.c
>> @@ -508,6 +508,9 @@ parse_link_speed(int n)
>> void
>> launch_args_parse(int argc, char** argv)
>> {
>> +#define PARAM_PROC_ID "proc-id"
>> +#define PARAM_NUM_PROCS "num-procs"
>> +
>> int n, opt;
>> char **argvopt;
>> int opt_idx;
>> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
>> { "rx-mq-mode", 1, 0, 0 },
>> { "record-core-cycles", 0, 0, 0 },
>> { "record-burst-stats", 0, 0, 0 },
>> + { PARAM_NUM_PROCS, 1, 0, 0 },
>> + { PARAM_PROC_ID, 1, 0, 0 },
>> { 0, 0, 0, 0 },
>> };
>> @@ -1391,6 +1396,13 @@ launch_args_parse(int argc, char** argv)
>> record_core_cycles = 1;
>> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
>> record_burst_stats = 1;
>> +
>
> To be consistent for rest, this empty line can be removed.
>
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_NUM_PROCS, 9) == 0)
>> + num_procs = atoi(optarg);
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_PROC_ID, 7) == 0)
>> + proc_id = atoi(optarg);
>> break;
>> case 'h':
>> usage(argv[0]);
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index 96d2e0f..01d0d82 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -518,6 +518,16 @@ enum rte_eth_rx_mq_mode rx_mq_mode =
>> ETH_MQ_RX_VMDQ_DCB_RSS;
>> */
>> uint32_t eth_link_speed;
>> +/*
>> + * Id of the current process.
>> + */
>
> Can you please add more info, like this being related to the
> primary/secondary process support and used to configure the queues to be
> polled.
>
>> +int proc_id;
>> +
>> +/*
>> + * Number of processes.
>> + */
>> +unsigned int num_procs = 1;
>> +
>
> Ditto.
>
> <...>
>
>> @@ -2511,21 +2537,28 @@ start_port(portid_t pid)
>> return -1;
>> }
>> /* configure port */
>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> + diag = rte_eth_dev_configure(pi,
>> + nb_rxq + nb_hairpinq,
>> nb_txq + nb_hairpinq,
>> &(port->dev_conf));
>> - if (diag != 0) {
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> - printf("Fail to configure port %d\n", pi);
>> - /* try to reconfigure port next time */
>> - port->need_reconfig = 1;
>> - return -1;
>> + if (diag != 0) {
>> + if (rte_atomic16_cmpset(
>> + &(port->port_status),
>> + RTE_PORT_HANDLING,
>> + RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set "
>> + "back to stopped\n", pi);
>> + printf("Fail to configure port %d\n",
>> + pi);
>> + /* try to reconfigure port next time */
>> + port->need_reconfig = 1;
>> + return -1;
>> + }
>
> Just an idea,
> I am a little worried about the complexity this new process type checks
> are added to the multiple locations. What do you think hiding process
> type checks behind new functions, like:
> eth_dev_configure_mp(...) {
> if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
> return rte_eth_dev_configure(..)
> }
> return 0;
> }
>
> and replace it with 'rte_eth_dev_configure()':
>
> -diag = rte_eth_dev_configure(...)
> +diag = eth_dev_configure_mp(...)
>
> Do you think does this help reducing the complexity added by the
> multi-process support?
>
>> }
>> }
>> - if (port->need_reconfig_queues > 0) {
>> + if (port->need_reconfig_queues > 0 &&
>> + rte_eal_process_type() == RTE_PROC_PRIMARY) {
>
> According our coding convention, we don't allign with paranthesis but
> put double tabs.
>
> And what about creating an inline function for primary process check,
> like is_proc_primary(), that may help.
>
>> port->need_reconfig_queues = 0;
>> /* setup tx queues */
>> for (qi = 0; qi < nb_txq; qi++) {
>> @@ -2626,17 +2659,20 @@ start_port(portid_t pid)
>> cnt_pi++;
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> - if (diag < 0) {
>> - printf("Fail to start port %d: %s\n", pi,
>> - rte_strerror(-diag));
>> + if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
>> + diag = rte_eth_dev_start(pi);
>> + if (diag < 0) {
>> + printf("Fail to start port %d: %s\n", pi,
>> + rte_strerror(-diag));
>> - /* Fail to setup rx queue, return */
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> + /* Fail to setup rx queue, return */
>> + if (rte_atomic16_cmpset(&(port->port_status),
>> RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>
> Syntax is wrong here after change, but if you go with above idea to hide
> process type check within functions, you won't need to change these
> lines at all.
>
> <...>
>
>> @@ -3885,8 +3925,10 @@ main(int argc, char** argv)
>> }
>> }
>> - if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
>> + if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
>> + pmd_test_exit();
>
> Why need to add 'pmd_test_exit()' for the multi-process support, if
> there is a special case, please add comment for it.
>
>
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v10] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (8 preceding siblings ...)
2021-04-16 1:52 ` [dpdk-dev] [PATCH v9] " Min Hu (Connor)
@ 2021-04-17 6:12 ` Min Hu (Connor)
2021-04-17 22:21 ` Ferruh Yigit
2021-04-19 1:03 ` [dpdk-dev] [PATCH v11] " Min Hu (Connor)
` (2 subsequent siblings)
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-17 6:12 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
---
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 21 +++++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 127 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++
7 files changed, 236 insertions(+), 25 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 5bf1497..e465824 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 40b2b29..d0e5b29 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2860,6 +2860,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2877,7 +2879,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2894,6 +2911,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..ece05c1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 96d2e0f..737a8e7 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,25 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2626,16 +2702,17 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
printf("Port %d can not be set back to "
- "stopped\n", pi);
+ "stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2842,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2911,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3100,7 +3179,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3431,7 +3510,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3621,6 +3700,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3786,10 +3869,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 36d8535..4c8f8bc 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -630,6 +630,15 @@ extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
extern enum rte_eth_rx_mq_mode rx_mq_mode;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 3bd7757..df01d4f 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -185,6 +185,7 @@ New Features
``show port (port_id) rxq (queue_id) desc used count``
* Added command to dump internal representation information of single flow.
``flow dump (port_id) rule (rule_id)``
+ * Added support multi-process for testpmd.
Removed Items
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index ec1dc7d..d714951 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -551,3 +551,89 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v10] app/testpmd: support multi-process
2021-04-17 6:12 ` [dpdk-dev] [PATCH v10] " Min Hu (Connor)
@ 2021-04-17 22:21 ` Ferruh Yigit
2021-04-19 1:03 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-17 22:21 UTC (permalink / raw)
To: Min Hu (Connor), dev
Cc: anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
On 4/17/2021 7:12 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Hi Connor,
I put some minor syntax comments, when they are fixed,
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
<...>
> if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> - printf("Fail to configure port %d\n", pi);
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set "
> + "back to stopped\n", pi);
It is OK to have long line for the strings, so it can be like
printf("Port %d can not be set back to stopped\n",
pi);
> + printf("Fail to configure port %d\n",
> + pi);
No need to update this line.
<...>
> /* Fail to setup rx queue, return */
> if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> printf("Port %d can not be set back to "
> - "stopped\n", pi);
> + "stopped\n",
> + pi);
These are syntax changes, I think can keep these lines as it is.
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v10] app/testpmd: support multi-process
2021-04-17 22:21 ` Ferruh Yigit
@ 2021-04-19 1:03 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-19 1:03 UTC (permalink / raw)
To: Ferruh Yigit, dev; +Cc: anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
Thanks Ferruh, fixed in v11, thanks.
在 2021/4/18 6:21, Ferruh Yigit 写道:
> On 4/17/2021 7:12 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>
> Hi Connor,
>
> I put some minor syntax comments, when they are fixed,
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> <...>
>
>> if (diag != 0) {
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> - printf("Fail to configure port %d\n", pi);
>> + if (rte_atomic16_cmpset(
>> + &(port->port_status),
>> + RTE_PORT_HANDLING,
>> + RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set "
>> + "back to stopped\n", pi);
>
> It is OK to have long line for the strings, so it can be like
> printf("Port %d can not be set back to stopped\n",
> pi);
>
>> + printf("Fail to configure port %d\n",
>> + pi);
>
> No need to update this line.
>
> <...>
>
>> /* Fail to setup rx queue, return */
>> if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> printf("Port %d can not be set back to "
>> - "stopped\n", pi);
>> + "stopped\n",
>> + pi);
>
> These are syntax changes, I think can keep these lines as it is.
>
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v11] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (9 preceding siblings ...)
2021-04-17 6:12 ` [dpdk-dev] [PATCH v10] " Min Hu (Connor)
@ 2021-04-19 1:03 ` Min Hu (Connor)
2021-04-19 13:42 ` Ferruh Yigit
2021-04-21 8:36 ` [dpdk-dev] [PATCH v12] " Min Hu (Connor)
2021-04-22 1:18 ` [dpdk-dev] [PATCH v13] " Min Hu (Connor)
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-19 1:03 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 21 +++++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 86 ++++++++++++++++++++++
7 files changed, 236 insertions(+), 27 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 5bf1497..e465824 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5354,6 +5354,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index d4b0e85..5d91335 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2859,6 +2859,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2876,7 +2878,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2893,6 +2910,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..ece05c1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index d4be23f..dc85e3a 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2548,8 +2623,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2626,16 +2701,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2909,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3101,7 +3178,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3787,10 +3868,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 36d8535..4c8f8bc 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -630,6 +630,15 @@ extern struct mplsoudp_decap_conf mplsoudp_decap_conf;
extern enum rte_eth_rx_mq_mode rx_mq_mode;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 58272e1..353889a 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -192,6 +192,7 @@ New Features
``show port (port_id) rxq (queue_id) desc used count``
* Added command to dump internal representation information of single flow.
``flow dump (port_id) rule (rule_id)``
+ * Added support multi-process for testpmd.
Removed Items
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index d062165..6642efc 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -543,3 +543,89 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+
+Testpmd Support Multi Process Command-line Options
+--------------------------------------------------
+
+The following are the command-line options for the testpmd applications(support
+multi process).They must be separated from the EAL options, shown in the previous
+section, with a ``--`` separator:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on proc_num and proc_id.
+Calculation rule for the Testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if supports 4 txq and rxq
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+Most dev ops is supported in primary and secondary process. While secondary process
+is not permitted to allocate or release shared memory, so some ops are not supported
+as follows:
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like:
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
+stats supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list(but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
+from others is not permitted.
+
+RSS supported, Primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary and secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v11] app/testpmd: support multi-process
2021-04-19 1:03 ` [dpdk-dev] [PATCH v11] " Min Hu (Connor)
@ 2021-04-19 13:42 ` Ferruh Yigit
2021-04-21 9:08 ` Min Hu (Connor)
0 siblings, 1 reply; 80+ messages in thread
From: Ferruh Yigit @ 2021-04-19 13:42 UTC (permalink / raw)
To: Min Hu (Connor), dev, John McNamara, Marko Kovacevic
Cc: anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
On 4/19/2021 2:03 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
Hi Connor,
I put some comments on the documentation, can you please check them?
Meanwhile, John, Marko can you please help on the doc review?
Thanks,
ferruh
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
<...>
> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> + diag = eth_dev_start_mp(pi);
> if (diag < 0) {
> printf("Fail to start port %d: %s\n", pi,
> rte_strerror(-diag));
>
> /* Fail to setup rx queue, return */
> if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> - "stopped\n", pi);
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set back to stopped\n",
> + pi);
Indentation is wrong.
<...>
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -543,3 +543,89 @@ The command line options are:
> bit 1 - two hairpin ports paired
> bit 0 - two hairpin ports loop
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +
> +Testpmd Support Multi Process Command-line Options
> +--------------------------------------------------
What do you think making this a sub-section of the "Testpmd Command-line Options"?
And the section name can be "Testpmd Multi-Process Command-line Options"
> +
> +The following are the command-line options for the testpmd applications(support
Space before '('.
> +multi process).They must be separated from the EAL options, shown in the previous
> +section, with a ``--`` separator:
If this becomes sub-section of the "Testpmd Command-line Options", above will
become duplication, and I think it can be simplified as:
"The following are the command-line options for testpmd multi-process support:"
> +
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=1
> +
> +The command line options are:
> +
> +* ``--rxq=N``
> +
> + Set the number of Rx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--txq=N``
> +
> + Set the number of Tx queues per port to N. N is the sum of queues used by primary
> + and secondary process. Primary process and secondary process should have separate
> + queues, and each should occupy at least one queue. Where N should be the multiple
> + of number of processes.
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary
> + process and secondary process.
> +
The famous question, does it start from '0' or '1'?
> +Calculation rule for queue:
> +All queues are allocated to different processes based on proc_num and proc_id.
Can highlight proc_num and proc_id, as ``proc_num`` and ``proc_id``.
> +Calculation rule for the Testpmd to allocate queues to each process:
Not sure if testpmd should start with uppercase.
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
Can you put above into a code-block.
> +For example, if supports 4 txq and rxq
".., if testpmp supports 4 Tx and Rx queues"
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +The number of rings should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. After RSS is
> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
> +Some traffic enters redundant queues and cannot be forwarded.
> +
> +Most dev ops is supported in primary and secondary process. While secondary process
"Most of the device operations are supported in ..."?
"While the secondary process ..."?
> +is not permitted to allocate or release shared memory, so some ops are not supported
> +as follows:
> +``dev_configure``
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
> +
This list displayed as single line in the html output, if the intention is to
have them listed as been in the source, may need following update:
-as follows:
+as follows::
+
> +So, any command from testpmd which calls those APIs will not be supported in secondary
> +process, like:
> +``port config all rxq|txq|rxd|txd <value>``
> +``port config <port_id> rx_offload xxx on/off ``
> +``port config <port_id> tx_offload xxx on/off``
> +etc.
Same here for the list view.
> +
> +RTE_FLOW supported, it applies only on its own process on SW side, but all on HW size.
You may use "Flow API", instead of RTE_FLOW, so it becomes: "Flow API is supported."
> +stats supported, stats will not change when one quit and start, As they share the same
> +buffer to store the stats.
The above stats related sentences looks like squeezed within the flow API
related description, it may be good to move it.
> Flow rules are maintained in process level: primary and secondary
> +has its own flow list(but one flow list in HW). The two can see all the queues, so setting
Space before '('.
> +the flow rules for the other is OK. Of course, io(receive or transmit packets) in the queue
> +from others is not permitted.
I understand what you mean, but wording can be improved, can you please try to
reword it?
> +
> +RSS supported, Primary process and secondary process has separate queues to use, RSS
"RSS is supported"? And start 'primary' with lowercase.
> +will work in their own queues whether primary and secondary process.
>
There is a chance that when RSS is enabled, the packet may end up in other
process' queues?
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v11] app/testpmd: support multi-process
2021-04-19 13:42 ` Ferruh Yigit
@ 2021-04-21 9:08 ` Min Hu (Connor)
0 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-21 9:08 UTC (permalink / raw)
To: Ferruh Yigit, dev, John McNamara, Marko Kovacevic
Cc: anatoly.burakov, andrew.rybchenko, thomas, xiaoyun.li
Hi, Ferruh,
在 2021/4/19 21:42, Ferruh Yigit 写道:
> On 4/19/2021 2:03 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>
> Hi Connor,
>
> I put some comments on the documentation, can you please check them?
>
> Meanwhile, John, Marko can you please help on the doc review?
>
> Thanks,
> ferruh
>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>
> <...>
>
>> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
>> cnt_pi++;
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> + diag = eth_dev_start_mp(pi);
>> if (diag < 0) {
>> printf("Fail to start port %d: %s\n", pi,
>> rte_strerror(-diag));
>> /* Fail to setup rx queue, return */
>> if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back to "
>> - "stopped\n", pi);
>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set back to stopped\n",
>> + pi);
>
> Indentation is wrong.
>
> <...>
>
>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>> @@ -543,3 +543,89 @@ The command line options are:
>> bit 1 - two hairpin ports paired
>> bit 0 - two hairpin ports loop
>> The default value is 0. Hairpin will use single port mode and
>> implicit Tx flow mode.
>> +
>> +
>> +Testpmd Support Multi Process Command-line Options
>> +--------------------------------------------------
>
> What do you think making this a sub-section of the "Testpmd Command-line
> Options"?
>
> And the section name can be "Testpmd Multi-Process Command-line Options"
>
>> +
>> +The following are the command-line options for the testpmd
>> applications(support
>
> Space before '('.
>
>> +multi process).They must be separated from the EAL options, shown in
>> the previous
>> +section, with a ``--`` separator:
>
> If this becomes sub-section of the "Testpmd Command-line Options", above
> will become duplication, and I think it can be simplified as:
>
> "The following are the command-line options for testpmd multi-process
> support:"
>
>
>> +
>> +.. code-block:: console
>> +
>> + primary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4
>> --txq=4 \
>> + --num-procs=2 --proc-id=0
>> +
>> + secondary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4
>> --txq=4 \
>> + --num-procs=2 --proc-id=1
>> +
>> +The command line options are:
>> +
>> +* ``--rxq=N``
>> +
>> + Set the number of Rx queues per port to N. N is the sum of queues
>> used by primary
>> + and secondary process. Primary process and secondary process
>> should have separate
>> + queues, and each should occupy at least one queue. Where N should
>> be the multiple
>> + of number of processes.
>> +
>> +* ``--txq=N``
>> +
>> + Set the number of Tx queues per port to N. N is the sum of queues
>> used by primary
>> + and secondary process. Primary process and secondary process
>> should have separate
>> + queues, and each should occupy at least one queue. Where N should
>> be the multiple
>> + of number of processes.
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
>> +* ``--proc-id=id``
>> +
>> + The id of the current process (id < num-procs). id should be
>> different in primary
>> + process and secondary process.
>> +
>
> The famous question, does it start from '0' or '1'?
>
>> +Calculation rule for queue:
>> +All queues are allocated to different processes based on proc_num and
>> proc_id.
>
> Can highlight proc_num and proc_id, as ``proc_num`` and ``proc_id``.
>
>> +Calculation rule for the Testpmd to allocate queues to each process:
>
> Not sure if testpmd should start with uppercase.
>
>> +start(queue start id) = proc_id * nb_q / num_procs；
>> +end(queue end id) = start + nb_q / num_procs；
>> +
>
> Can you put above into a code-block.
>
Have already put above into a code-block in previous version.
>> +For example, if supports 4 txq and rxq
>
> ".., if testpmp supports 4 Tx and Rx queues"
>
>> +the 0~1 for primary process
>> +the 2~3 for secondary process
>> +
>> +The number of rings should be a multiple of the number of processes.
>> If not,
>> +redundant queues will exist after queues are allocated to processes.
>> After RSS is
>> +enabled, packet loss occurs when traffic is sent to all processes at
>> the same time.
>> +Some traffic enters redundant queues and cannot be forwarded.
>> +
>> +Most dev ops is supported in primary and secondary process. While
>> secondary process
>
> "Most of the device operations are supported in ..."?
> "While the secondary process ..."?
>
>> +is not permitted to allocate or release shared memory, so some ops
>> are not supported
>> +as follows:
>> +``dev_configure``
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
>> +
>
> This list displayed as single line in the html output, if the intention
> is to have them listed as been in the source, may need following update:
>
> -as follows:
> +as follows::
> +
>
>> +So, any command from testpmd which calls those APIs will not be
>> supported in secondary
>> +process, like:
>> +``port config all rxq|txq|rxd|txd <value>``
>> +``port config <port_id> rx_offload xxx on/off ``
>> +``port config <port_id> tx_offload xxx on/off``
>> +etc.
>
> Same here for the list view.
>
>> +
>> +RTE_FLOW supported, it applies only on its own process on SW side,
>> but all on HW size.
>
> You may use "Flow API", instead of RTE_FLOW, so it becomes: "Flow API is
> supported."
>
>> +stats supported, stats will not change when one quit and start, As
>> they share the same
>> +buffer to store the stats.
>
> The above stats related sentences looks like squeezed within the flow
> API related description, it may be good to move it.
>
>> Flow rules are maintained in process level: primary and secondary
>> +has its own flow list(but one flow list in HW). The two can see all
>> the queues, so setting
>
> Space before '('.
>
>> +the flow rules for the other is OK. Of course, io(receive or transmit
>> packets) in the queue
>> +from others is not permitted.
>
> I understand what you mean, but wording can be improved, can you please
> try to reword it?
>
>> +
>> +RSS supported, Primary process and secondary process has separate
>> queues to use, RSS
>
> "RSS is supported"? And start 'primary' with lowercase.
>
>> +will work in their own queues whether primary and secondary process.
>>
>
> There is a chance that when RSS is enabled, the packet may end up in
> other process' queues?
When RSS is enabled, packet could end up in other process's queues, but
primary and secondary receives packerts from their own queues, and
they do not interfere with each other, do not conflict with each other.
> .
^ permalink raw reply [flat|nested] 80+ messages in thread
* [dpdk-dev] [PATCH v12] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (10 preceding siblings ...)
2021-04-19 1:03 ` [dpdk-dev] [PATCH v11] " Min Hu (Connor)
@ 2021-04-21 8:36 ` Min Hu (Connor)
2021-04-22 1:18 ` [dpdk-dev] [PATCH v13] " Min Hu (Connor)
12 siblings, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-21 8:36 UTC (permalink / raw)
To: dev
Cc: ferruh.yigit, john.mcnamara, marko.kovacevic, anatoly.burakov,
andrew.rybchenko, thomas, xiaoyun.li
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 21 +++++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 84 +++++++++++++++++++++
7 files changed, 234 insertions(+), 27 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ccaeefa..b6bf4cd 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5357,6 +5357,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ccb9bd3..3c7a378 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2924,6 +2924,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2941,7 +2943,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2958,6 +2975,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..ece05c1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index d4be23f..afa2a6b 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2548,8 +2623,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2626,16 +2701,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2909,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3101,7 +3178,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3787,10 +3868,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 9530ec5..e2939ea 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 668fca8..cc88aea 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -230,6 +230,7 @@ New Features
* Added commands to construct conntrack context and relevant indirect
action handle creation, update for conntrack action as well as conntrack
item matching.
+ * Added support multi-process for testpmd.
* **Updated ipsec-secgw sample application.**
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index d062165..7414e85 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -543,3 +543,87 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if testpmp supports 4 Tx and Rx queues
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows::
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like::
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+Flow API is supported, it applies only on its own process on SW side, but all on HW size.
+
+Stats is supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list (but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. But in testpmd primary process receives or transmits
+packets from the queue allocated for secondary process is not permitted, and same for
+secondary process.
+
+RSS is supported, primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary or secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
` (11 preceding siblings ...)
2021-04-21 8:36 ` [dpdk-dev] [PATCH v12] " Min Hu (Connor)
@ 2021-04-22 1:18 ` Min Hu (Connor)
2021-06-08 8:42 ` Andrew Rybchenko
12 siblings, 1 reply; 80+ messages in thread
From: Min Hu (Connor) @ 2021-04-22 1:18 UTC (permalink / raw)
To: dev; +Cc: ferruh.yigit, thomas, aman.deep.singh
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v13:
* Modified the doc syntax.
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 21 +++++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 70 ++++++++++++++++++
7 files changed, 220 insertions(+), 27 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 12efbc0..f0fa6e8 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5450,6 +5450,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index e189062..9eb1fa7 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2971,6 +2971,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2988,7 +2990,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -3005,6 +3022,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..ece05c1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index d4be23f..afa2a6b 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2548,8 +2623,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2626,16 +2701,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2909,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3101,7 +3178,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3787,10 +3868,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 6ca872d..3318e8f 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index b36c120..9361332 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -235,6 +235,7 @@ New Features
``port cleanup (port_id) txq (queue_id) (free_cnt)``
* Added command to show link flow control info.
``show port (port_id) flow_ctrl``
+ * Added support multi-process for testpmd.
* **Updated ipsec-secgw sample application.**
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index d062165..74efa4f 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -543,3 +543,73 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs；
+end(queue end id) = start + nb_q / num_procs；
+
+For example, if testpmd supports 4 Tx and Rx queues
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows::
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like::
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+Flow API is supported, it applies only on its own process on SW side, but all on HW side.
+
+Stats is supported, stats will not change when one quit and start, as they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list (but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. But in the testpmd primary process receiving or transmitting
+packets from the queue allocated for secondary process is not permitted, and same for
+secondary process.
+
+RSS is supported, primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary or secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-04-22 1:18 ` [dpdk-dev] [PATCH v13] " Min Hu (Connor)
@ 2021-06-08 8:42 ` Andrew Rybchenko
2021-06-08 10:22 ` Thomas Monjalon
2021-06-15 12:04 ` Min Hu (Connor)
0 siblings, 2 replies; 80+ messages in thread
From: Andrew Rybchenko @ 2021-06-08 8:42 UTC (permalink / raw)
To: Min Hu (Connor), dev; +Cc: ferruh.yigit, thomas, aman.deep.singh
@Thomas, @Ferruh, please, see question below.
On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
> This patch adds multi-process support for testpmd.
> The test cmd example as follows:
> the primary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
> Signed-off-by: Lijun Ou <oulijun@huawei.com>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> v13:
> * Modified the doc syntax.
>
> v12:
> * Updated doc info.
>
> v11:
> * Fixed some minor syntax.
>
> v10:
> * Hid process type checks behind new functions.
> * Added comments.
>
> v9:
> * Updated release notes and rst doc.
> * Deleted deprecated codes.
> * move macro and variable.
>
> v8:
> * Added warning info about queue numbers and process numbers.
>
> v7:
> * Fixed compiling error for unexpected unindent.
>
> v6:
> * Add rte flow description for multiple process.
>
> v5:
> * Fixed run_app.rst for multiple process description.
> * Fix compiling error.
>
> v4:
> * Fixed minimum vlaue of Rxq or Txq in doc.
>
> v3:
> * Fixed compiling error using gcc10.0.
>
> v2:
> * Added document for this patch.
> ---
> app/test-pmd/cmdline.c | 6 ++
> app/test-pmd/config.c | 21 +++++-
> app/test-pmd/parameters.c | 11 +++
> app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
> app/test-pmd/testpmd.h | 9 +++
> doc/guides/rel_notes/release_21_05.rst | 1 +
> doc/guides/testpmd_app_ug/run_app.rst | 70 ++++++++++++++++++
> 7 files changed, 220 insertions(+), 27 deletions(-)
>
> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
> index 12efbc0..f0fa6e8 100644
> --- a/app/test-pmd/cmdline.c
> +++ b/app/test-pmd/cmdline.c
> @@ -5450,6 +5450,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
> __rte_unused void *data)
> {
> struct cmd_set_flush_rx *res = parsed_result;
> +
> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
> + printf("multi-process doesn't support to flush rx queues.\n");
rx -> Rx
> + return;
> + }
> +
> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
> }
>
> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
> index e189062..9eb1fa7 100644
> --- a/app/test-pmd/config.c
> +++ b/app/test-pmd/config.c
> @@ -2971,6 +2971,8 @@ rss_fwd_config_setup(void)
> queueid_t rxq;
> queueid_t nb_q;
> streamid_t sm_id;
> + int start;
> + int end;
>
> nb_q = nb_rxq;
> if (nb_q > nb_txq)
> @@ -2988,7 +2990,22 @@ rss_fwd_config_setup(void)
> init_fwd_streams();
>
> setup_fwd_config_of_each_lcore(&cur_fwd_config);
> - rxp = 0; rxq = 0;
> +
> + if (proc_id > 0 && nb_q % num_procs)
Please, compare result with 0 explicitly.
> + printf("Warning! queue numbers should be multiple of "
> + "processes, or packet loss will happen.\n");
Do not split format string across multiple lines.
Frankly speaking I don't undertand why. Why is it impossible to
serve 2 queues in the first process and 1 queue in the second
process if 3 queues and 2 processes are configured.
I think RSS redirection table can perfectly do it.
> +
> + /**
> + * In multi-process, All queues are allocated to different
> + * processes based on num_procs and proc_id. For example:
> + * if supports 4 queues(nb_q), 2 processes(num_procs),
> + * the 0~1 queue for primary process.
> + * the 2~3 queue for secondary process.
> + */
> + start = proc_id * nb_q / num_procs;
> + end = start + nb_q / num_procs;
> + rxp = 0;
> + rxq = start;
> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
> struct fwd_stream *fs;
>
> @@ -3005,6 +3022,8 @@ rss_fwd_config_setup(void)
> continue;
> rxp = 0;
> rxq++;
> + if (rxq >= end)
> + rxq = start;
> }
> }
>
> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
> index f3954c1..ece05c1 100644
> --- a/app/test-pmd/parameters.c
> +++ b/app/test-pmd/parameters.c
> @@ -508,6 +508,9 @@ parse_link_speed(int n)
> void
> launch_args_parse(int argc, char** argv)
> {
> +#define PARAM_PROC_ID "proc-id"
> +#define PARAM_NUM_PROCS "num-procs"
> +
> int n, opt;
> char **argvopt;
> int opt_idx;
> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
> { "rx-mq-mode", 1, 0, 0 },
> { "record-core-cycles", 0, 0, 0 },
> { "record-burst-stats", 0, 0, 0 },
> + { PARAM_NUM_PROCS, 1, 0, 0 },
> + { PARAM_PROC_ID, 1, 0, 0 },
> { 0, 0, 0, 0 },
> };
>
> @@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
> record_core_cycles = 1;
> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
> record_burst_stats = 1;
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_NUM_PROCS, 9) == 0)
I strongly dislike 9 here and 7 below. Why is strncmp() used
here, but just strcmp() is used for all other options.
It makes the code inconsistent.
> + num_procs = atoi(optarg);
> + if (strncmp(lgopts[opt_idx].name,
> + PARAM_PROC_ID, 7) == 0)
> + proc_id = atoi(optarg);
> break;
> case 'h':
> usage(argv[0]);
> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
> index d4be23f..afa2a6b 100644
> --- a/app/test-pmd/testpmd.c
> +++ b/app/test-pmd/testpmd.c
> @@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
> */
> uint32_t eth_link_speed;
>
> +/*
> + * Id of the current process in multi-process, used to
Id -> ID in accordance with devtools/words-case.txt
> + * configure the queues to be polled.
> + */
> +int proc_id;
> +
> +/*
> + * Number of processes in multi-process, used to
> + * configure the queues to be polled.
> + */
> +unsigned int num_procs = 1;
> +
> +static int
> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
> + const struct rte_eth_conf *dev_conf)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
> + dev_conf);
> + return 0;
> +}
> +
> +static int
> +eth_dev_start_mp(uint16_t port_id)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_start(port_id);
> +
> + return 0;
> +}
> +
> +static int
> +eth_dev_stop_mp(uint16_t port_id)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_stop(port_id);
> +
> + return 0;
> +}
> +
> +static void
> +mempool_free_mp(struct rte_mempool *mp)
> +{
> + if (is_proc_primary())
> + return rte_mempool_free(mp);
As far as I remember some compilers do not like it for void.
Just remove 'return'.
> +}
> +
> +static int
> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
> +{
> + if (is_proc_primary())
> + return rte_eth_dev_set_mtu(port_id, mtu);
> +
> + return 0;
> +}
> +
> /* Forward function declarations */
> static void setup_attached_port(portid_t pi);
> static void check_all_ports_link_status(uint32_t port_mask);
> @@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
> mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
> mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
>
> + if (!is_proc_primary()) {
> + rte_mp = rte_mempool_lookup(pool_name);
> + goto err;
It looks like error path, but it works in the case of success
as well. Looks confusing.
> + }
> +
> TESTPMD_LOG(INFO,
> "create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
> pool_name, nb_mbuf, mbuf_seg_size, socket_id);
> @@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
>
> err:
> if (rte_mp == NULL) {
> - rte_exit(EXIT_FAILURE,
> - "Creation of mbuf pool for socket %u failed: %s\n",
> - socket_id, rte_strerror(rte_errno));
> + if (is_proc_primary())
> + rte_exit(EXIT_FAILURE,
> + "Creation of mbuf pool for socket %u failed: %s\n",
> + socket_id, rte_strerror(rte_errno));
> + else
> + rte_exit(EXIT_FAILURE,
> + "Get mbuf pool for socket %u failed: %s\n",
> + socket_id, rte_strerror(rte_errno));
> } else if (verbose_level > 0) {
> rte_mempool_dump(stdout, rte_mp);
> }
> @@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
> uint64_t timer_period;
>
> + if (num_procs > 1) {
> + printf("multi-process not support for flushing fwd rx "
rx -> Rx
also it is better to avoid like split.
> + "queues, skip the below lines and return.\n");
> + return;
> + }
> +
> /* convert to number of cycles */
> timer_period = rte_get_timer_hz(); /* 1 second timeout */
>
> @@ -2511,21 +2583,24 @@ start_port(portid_t pid)
> return -1;
> }
> /* configure port */
> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
> - nb_txq + nb_hairpinq,
> - &(port->dev_conf));
> + diag = eth_dev_configure_mp(pi,
> + nb_rxq + nb_hairpinq,
> + nb_txq + nb_hairpinq,
> + &(port->dev_conf));
> if (diag != 0) {
> - if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> + if (rte_atomic16_cmpset(
> + &(port->port_status),
> + RTE_PORT_HANDLING,
> + RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set back to stopped\n",
can not -> cannot (since you touch the line anyway)
> + pi);
> printf("Fail to configure port %d\n", pi);
> /* try to reconfigure port next time */
> port->need_reconfig = 1;
> return -1;
> }
> }
> - if (port->need_reconfig_queues > 0) {
> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
> port->need_reconfig_queues = 0;
> /* setup tx queues */
> for (qi = 0; qi < nb_txq; qi++) {
> @@ -2548,8 +2623,8 @@ start_port(portid_t pid)
> if (rte_atomic16_cmpset(&(port->port_status),
> RTE_PORT_HANDLING,
> RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back "
> - "to stopped\n", pi);
> + printf("Port %d can not be set back to stopped\n",
can not -> cannot
> + pi);
> printf("Fail to configure port %d tx queues\n",
> pi);
> /* try to reconfigure queues next time */
> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
> cnt_pi++;
>
> /* start port */
> - diag = rte_eth_dev_start(pi);
> + diag = eth_dev_start_mp(pi);
> if (diag < 0) {
> printf("Fail to start port %d: %s\n", pi,
> rte_strerror(-diag));
>
> /* Fail to setup rx queue, return */
> if (rte_atomic16_cmpset(&(port->port_status),
> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> - printf("Port %d can not be set back to "
> - "stopped\n", pi);
> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
> + printf("Port %d can not be set back to stopped\n",
can not -> cannot
> + pi);
> continue;
> }
>
> @@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
> if (port->flow_list)
> port_flow_flush(pi);
>
> - if (rte_eth_dev_stop(pi) != 0)
> + if (eth_dev_stop_mp(pi) != 0)
> RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
> pi);
>
> @@ -2834,8 +2909,10 @@ close_port(portid_t pid)
> continue;
> }
>
> - port_flow_flush(pi);
> - rte_eth_dev_close(pi);
> + if (is_proc_primary()) {
> + port_flow_flush(pi);
> + rte_eth_dev_close(pi);
> + }
> }
>
> remove_invalid_ports();
> @@ -3101,7 +3178,7 @@ pmd_test_exit(void)
> }
> for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
> if (mempools[i])
> - rte_mempool_free(mempools[i]);
> + mempool_free_mp(mempools[i]);
> }
>
> printf("\nBye...\n");
> @@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
> * if unset do it here
> */
> if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
> - ret = rte_eth_dev_set_mtu(portid,
> + ret = eth_dev_set_mtu_mp(portid,
> port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
> if (ret)
> printf("Failed to set MTU to %u for port %u\n",
> @@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
> int retval;
> uint16_t i;
>
> + if (num_procs > 1) {
> + printf("The multi-process feature doesn't support dcb.\n");
> + return -ENOTSUP;
> + }
> rte_port = &ports[pid];
>
> memset(&port_conf, 0, sizeof(struct rte_eth_conf));
> @@ -3787,10 +3868,6 @@ main(int argc, char** argv)
> rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
> rte_strerror(rte_errno));
>
> - if (rte_eal_process_type() == RTE_PROC_SECONDARY)
> - rte_exit(EXIT_FAILURE,
> - "Secondary process type not supported.\n");
> -
> ret = register_eth_event_callback();
> if (ret != 0)
> rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
> index 6ca872d..3318e8f 100644
> --- a/app/test-pmd/testpmd.h
> +++ b/app/test-pmd/testpmd.h
> @@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
>
> extern struct rte_flow_action_conntrack conntrack_context;
>
> +extern int proc_id;
> +extern unsigned int num_procs;
> +
> +static inline bool
> +is_proc_primary(void)
> +{
> + return rte_eal_process_type() == RTE_PROC_PRIMARY;
> +}
> +
> static inline unsigned int
> lcore_num(void)
> {
> diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
> index b36c120..9361332 100644
> --- a/doc/guides/rel_notes/release_21_05.rst
> +++ b/doc/guides/rel_notes/release_21_05.rst
> @@ -235,6 +235,7 @@ New Features
> ``port cleanup (port_id) txq (queue_id) (free_cnt)``
> * Added command to show link flow control info.
> ``show port (port_id) flow_ctrl``
> + * Added support multi-process for testpmd.
Please, rebase to 21.08.
>
> * **Updated ipsec-secgw sample application.**
>
> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
> index d062165..74efa4f 100644
> --- a/doc/guides/testpmd_app_ug/run_app.rst
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> @@ -543,3 +543,73 @@ The command line options are:
> bit 1 - two hairpin ports paired
> bit 0 - two hairpin ports loop
> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
> +
> +Testpmd Multi-Process Command-line Options
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The following are the command-line options for testpmd multi-process support:
> +
> +.. code-block:: console
> +
> + primary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=0
> +
> + secondary process:
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
> + --num-procs=2 --proc-id=1
Do we really need --rxq=4 --txq=4 in secodary processes?
Can't we get it from already configured device in primary
process?
Same question is applicable to --num-procs. May be testpmd
can publish in shared memory? If yes, I'm OK to either
do it in the patch or defer as a later improvement.
> +
> +The command line options are:
> +
> +* ``--num-procs=N``
> +
> + The number of processes which will be used.
> +
> +* ``--proc-id=id``
> +
> + The id of the current process (id < num-procs). id should be different in primary
> + process and secondary process, which starts from '0'.
> +
> +Calculation rule for queue:
> +All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
> +Calculation rule for the testpmd to allocate queues to each process:
> +start(queue start id) = proc_id * nb_q / num_procs；
> +end(queue end id) = start + nb_q / num_procs；
> +
> +For example, if testpmd supports 4 Tx and Rx queues
> +the 0~1 for primary process
> +the 2~3 for secondary process
> +
> +The number of rings should be a multiple of the number of processes. If not,
> +redundant queues will exist after queues are allocated to processes. After RSS is
> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
> +Some traffic enters redundant queues and cannot be forwarded.
> +
> +All the dev ops is supported in primary process. While secondary process is not permitted
> +to allocate or release shared memory, so some ops are not supported as follows::
> +``dev_configure``
> +``dev_start``
> +``dev_stop``
> +``rx_queue_setup``
> +``tx_queue_setup``
> +``rx_queue_release``
> +``tx_queue_release``
@Thomas, @Ferrh, shouldn't it be handled on ethdev level as
well if it is really that strict.
> +
> +So, any command from testpmd which calls those APIs will not be supported in secondary
> +process, like::
> +``port config all rxq|txq|rxd|txd <value>``
> +``port config <port_id> rx_offload xxx on/off ``
> +``port config <port_id> tx_offload xxx on/off``
> +etc.
> +
> +Flow API is supported, it applies only on its own process on SW side, but all on HW side.
Sorry, I don't understand it.
> +
> +Stats is supported, stats will not change when one quit and start, as they share the same
> +buffer to store the stats. Flow rules are maintained in process level: primary and secondary
> +has its own flow list (but one flow list in HW). The two can see all the queues, so setting
> +the flow rules for the other is OK. But in the testpmd primary process receiving or transmitting
> +packets from the queue allocated for secondary process is not permitted, and same for
> +secondary process.
> +
> +RSS is supported, primary process and secondary process has separate queues to use, RSS
> +will work in their own queues whether primary or secondary process.
For me it sounds like secondary process has own RSS
configuration. If so, it is false. I guess I simply
misunderstand above paragraph.
^ permalink raw reply [flat|nested] 80+ messages in thread* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-06-08 8:42 ` Andrew Rybchenko
@ 2021-06-08 10:22 ` Thomas Monjalon
2021-06-08 10:39 ` Andrew Rybchenko
2021-06-15 12:04 ` Min Hu (Connor)
1 sibling, 1 reply; 80+ messages in thread
From: Thomas Monjalon @ 2021-06-08 10:22 UTC (permalink / raw)
To: Min Hu (Connor), Andrew Rybchenko; +Cc: dev, ferruh.yigit, aman.deep.singh
08/06/2021 10:42, Andrew Rybchenko:
> On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
> > --- a/doc/guides/testpmd_app_ug/run_app.rst
> > +++ b/doc/guides/testpmd_app_ug/run_app.rst
[...]
> > +All the dev ops is supported in primary process. While secondary process is not permitted
> > +to allocate or release shared memory, so some ops are not supported as follows::
> > +``dev_configure``
> > +``dev_start``
> > +``dev_stop``
> > +``rx_queue_setup``
> > +``tx_queue_setup``
> > +``rx_queue_release``
> > +``tx_queue_release``
>
> @Thomas, @Ferrh, shouldn't it be handled on ethdev level as
> well if it is really that strict.
Yes it should be documented at ethdev level, not testpmd.
I think it was kept fuzzy for too long.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-06-08 10:22 ` Thomas Monjalon
@ 2021-06-08 10:39 ` Andrew Rybchenko
2021-06-08 12:02 ` Thomas Monjalon
2021-06-08 12:36 ` Ferruh Yigit
0 siblings, 2 replies; 80+ messages in thread
From: Andrew Rybchenko @ 2021-06-08 10:39 UTC (permalink / raw)
To: Thomas Monjalon, Min Hu (Connor); +Cc: dev, ferruh.yigit, aman.deep.singh
On 6/8/21 1:22 PM, Thomas Monjalon wrote:
> 08/06/2021 10:42, Andrew Rybchenko:
>> On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
>>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> [...]
>>> +All the dev ops is supported in primary process. While secondary process is not permitted
>>> +to allocate or release shared memory, so some ops are not supported as follows::
>>> +``dev_configure``
>>> +``dev_start``
>>> +``dev_stop``
>>> +``rx_queue_setup``
>>> +``tx_queue_setup``
>>> +``rx_queue_release``
>>> +``tx_queue_release``
>>
>> @Thomas, @Ferrh, shouldn't it be handled on ethdev level as
>> well if it is really that strict.
>
> Yes it should be documented at ethdev level, not testpmd.
> I think it was kept fuzzy for too long.
To document is good, but I'm talking about more -
add checks in corresponding API functions and
return error.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-06-08 10:39 ` Andrew Rybchenko
@ 2021-06-08 12:02 ` Thomas Monjalon
2021-06-08 12:36 ` Ferruh Yigit
1 sibling, 0 replies; 80+ messages in thread
From: Thomas Monjalon @ 2021-06-08 12:02 UTC (permalink / raw)
To: Min Hu (Connor), Andrew Rybchenko; +Cc: dev, ferruh.yigit, aman.deep.singh
08/06/2021 12:39, Andrew Rybchenko:
> On 6/8/21 1:22 PM, Thomas Monjalon wrote:
> > 08/06/2021 10:42, Andrew Rybchenko:
> >> On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
> >>> --- a/doc/guides/testpmd_app_ug/run_app.rst
> >>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
> > [...]
> >>> +All the dev ops is supported in primary process. While secondary process is not permitted
> >>> +to allocate or release shared memory, so some ops are not supported as follows::
> >>> +``dev_configure``
> >>> +``dev_start``
> >>> +``dev_stop``
> >>> +``rx_queue_setup``
> >>> +``tx_queue_setup``
> >>> +``rx_queue_release``
> >>> +``tx_queue_release``
> >>
> >> @Thomas, @Ferrh, shouldn't it be handled on ethdev level as
> >> well if it is really that strict.
> >
> > Yes it should be documented at ethdev level, not testpmd.
> > I think it was kept fuzzy for too long.
>
> To document is good, but I'm talking about more -
> add checks in corresponding API functions and
> return error.
Yes I am OK with adding checks of the running process.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-06-08 10:39 ` Andrew Rybchenko
2021-06-08 12:02 ` Thomas Monjalon
@ 2021-06-08 12:36 ` Ferruh Yigit
1 sibling, 0 replies; 80+ messages in thread
From: Ferruh Yigit @ 2021-06-08 12:36 UTC (permalink / raw)
To: Andrew Rybchenko, Thomas Monjalon, Min Hu (Connor)
Cc: dev, aman.deep.singh, Anatoly Burakov
On 6/8/2021 11:39 AM, Andrew Rybchenko wrote:
> On 6/8/21 1:22 PM, Thomas Monjalon wrote:
>> 08/06/2021 10:42, Andrew Rybchenko:
>>> On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
>>>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>>>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>> [...]
>>>> +All the dev ops is supported in primary process. While secondary process is not permitted
>>>> +to allocate or release shared memory, so some ops are not supported as follows::
>>>> +``dev_configure``
>>>> +``dev_start``
>>>> +``dev_stop``
>>>> +``rx_queue_setup``
>>>> +``tx_queue_setup``
>>>> +``rx_queue_release``
>>>> +``tx_queue_release``
>>>
>>> @Thomas, @Ferrh, shouldn't it be handled on ethdev level as
>>> well if it is really that strict.
>>
>> Yes it should be documented at ethdev level, not testpmd.
>> I think it was kept fuzzy for too long.
>
> To document is good, but I'm talking about more -
> add checks in corresponding API functions and
> return error.
>
+Anatoly as multi process maintainer.
There are already some PMDs have these checks in the PMD level.
But current approach is providing more flexibility, if application takes care of
the synchronization, is there any reason some of above APIs can't be used by the
secondary process?
I am not sure if this flexibility is used or needed, if not needed +1 have the
checks in the ethdev layer and remove the PMD ones.
^ permalink raw reply [flat|nested] 80+ messages in thread
* Re: [dpdk-dev] [PATCH v13] app/testpmd: support multi-process
2021-06-08 8:42 ` Andrew Rybchenko
2021-06-08 10:22 ` Thomas Monjalon
@ 2021-06-15 12:04 ` Min Hu (Connor)
1 sibling, 0 replies; 80+ messages in thread
From: Min Hu (Connor) @ 2021-06-15 12:04 UTC (permalink / raw)
To: Andrew Rybchenko, dev; +Cc: ferruh.yigit, thomas, aman.deep.singh
Hi, Andrew,
see replies below, and others without no reply will be fixed in v14.
在 2021/6/8 16:42, Andrew Rybchenko 写道:
> @Thomas, @Ferruh, please, see question below.
>
> On 4/22/21 4:18 AM, Min Hu (Connor) wrote:
>> This patch adds multi-process support for testpmd.
>> The test cmd example as follows:
>> the primary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>
>> the secondary cmd:
>> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>
>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> v13:
>> * Modified the doc syntax.
>>
>> v12:
>> * Updated doc info.
>>
>> v11:
>> * Fixed some minor syntax.
>>
>> v10:
>> * Hid process type checks behind new functions.
>> * Added comments.
>>
>> v9:
>> * Updated release notes and rst doc.
>> * Deleted deprecated codes.
>> * move macro and variable.
>>
>> v8:
>> * Added warning info about queue numbers and process numbers.
>>
>> v7:
>> * Fixed compiling error for unexpected unindent.
>>
>> v6:
>> * Add rte flow description for multiple process.
>>
>> v5:
>> * Fixed run_app.rst for multiple process description.
>> * Fix compiling error.
>>
>> v4:
>> * Fixed minimum vlaue of Rxq or Txq in doc.
>>
>> v3:
>> * Fixed compiling error using gcc10.0.
>>
>> v2:
>> * Added document for this patch.
>> ---
>> app/test-pmd/cmdline.c | 6 ++
>> app/test-pmd/config.c | 21 +++++-
>> app/test-pmd/parameters.c | 11 +++
>> app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
>> app/test-pmd/testpmd.h | 9 +++
>> doc/guides/rel_notes/release_21_05.rst | 1 +
>> doc/guides/testpmd_app_ug/run_app.rst | 70 ++++++++++++++++++
>> 7 files changed, 220 insertions(+), 27 deletions(-)
>>
>> diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
>> index 12efbc0..f0fa6e8 100644
>> --- a/app/test-pmd/cmdline.c
>> +++ b/app/test-pmd/cmdline.c
>> @@ -5450,6 +5450,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
>> __rte_unused void *data)
>> {
>> struct cmd_set_flush_rx *res = parsed_result;
>> +
>> + if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
>> + printf("multi-process doesn't support to flush rx queues.\n");
>
> rx -> Rx
>
>> + return;
>> + }
>> +
>> no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
>> }
>>
>> diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
>> index e189062..9eb1fa7 100644
>> --- a/app/test-pmd/config.c
>> +++ b/app/test-pmd/config.c
>> @@ -2971,6 +2971,8 @@ rss_fwd_config_setup(void)
>> queueid_t rxq;
>> queueid_t nb_q;
>> streamid_t sm_id;
>> + int start;
>> + int end;
>>
>> nb_q = nb_rxq;
>> if (nb_q > nb_txq)
>> @@ -2988,7 +2990,22 @@ rss_fwd_config_setup(void)
>> init_fwd_streams();
>>
>> setup_fwd_config_of_each_lcore(&cur_fwd_config);
>> - rxp = 0; rxq = 0;
>> +
>> + if (proc_id > 0 && nb_q % num_procs)
>
> Please, compare result with 0 explicitly.
>
>> + printf("Warning! queue numbers should be multiple of "
>> + "processes, or packet loss will happen.\n");
>
> Do not split format string across multiple lines.
>
> Frankly speaking I don't undertand why. Why is it impossible to
> serve 2 queues in the first process and 1 queue in the second
> process if 3 queues and 2 processes are configured.
> I think RSS redirection table can perfectly do it.
>
Well, currently, my patch is one design implementation. I think
this can be done for later improvemnet.
>> +
>> + /**
>> + * In multi-process, All queues are allocated to different
>> + * processes based on num_procs and proc_id. For example:
>> + * if supports 4 queues(nb_q), 2 processes(num_procs),
>> + * the 0~1 queue for primary process.
>> + * the 2~3 queue for secondary process.
>> + */
>> + start = proc_id * nb_q / num_procs;
>> + end = start + nb_q / num_procs;
>> + rxp = 0;
>> + rxq = start;
>> for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
>> struct fwd_stream *fs;
>>
>> @@ -3005,6 +3022,8 @@ rss_fwd_config_setup(void)
>> continue;
>> rxp = 0;
>> rxq++;
>> + if (rxq >= end)
>> + rxq = start;
>> }
>> }
>>
>> diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
>> index f3954c1..ece05c1 100644
>> --- a/app/test-pmd/parameters.c
>> +++ b/app/test-pmd/parameters.c
>> @@ -508,6 +508,9 @@ parse_link_speed(int n)
>> void
>> launch_args_parse(int argc, char** argv)
>> {
>> +#define PARAM_PROC_ID "proc-id"
>> +#define PARAM_NUM_PROCS "num-procs"
>> +
>> int n, opt;
>> char **argvopt;
>> int opt_idx;
>> @@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
>> { "rx-mq-mode", 1, 0, 0 },
>> { "record-core-cycles", 0, 0, 0 },
>> { "record-burst-stats", 0, 0, 0 },
>> + { PARAM_NUM_PROCS, 1, 0, 0 },
>> + { PARAM_PROC_ID, 1, 0, 0 },
>> { 0, 0, 0, 0 },
>> };
>>
>> @@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
>> record_core_cycles = 1;
>> if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
>> record_burst_stats = 1;
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_NUM_PROCS, 9) == 0)
>
> I strongly dislike 9 here and 7 below. Why is strncmp() used
> here, but just strcmp() is used for all other options.
> It makes the code inconsistent.
>
>> + num_procs = atoi(optarg);
>> + if (strncmp(lgopts[opt_idx].name,
>> + PARAM_PROC_ID, 7) == 0)
>> + proc_id = atoi(optarg);
>> break;
>> case 'h':
>> usage(argv[0]);
>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>> index d4be23f..afa2a6b 100644
>> --- a/app/test-pmd/testpmd.c
>> +++ b/app/test-pmd/testpmd.c
>> @@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
>> */
>> uint32_t eth_link_speed;
>>
>> +/*
>> + * Id of the current process in multi-process, used to
>
> Id -> ID in accordance with devtools/words-case.txt
>
>> + * configure the queues to be polled.
>> + */
>> +int proc_id;
>> +
>> +/*
>> + * Number of processes in multi-process, used to
>> + * configure the queues to be polled.
>> + */
>> +unsigned int num_procs = 1;
>> +
>> +static int
>> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
>> + const struct rte_eth_conf *dev_conf)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
>> + dev_conf);
>> + return 0;
>> +}
>> +
>> +static int
>> +eth_dev_start_mp(uint16_t port_id)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_start(port_id);
>> +
>> + return 0;
>> +}
>> +
>> +static int
>> +eth_dev_stop_mp(uint16_t port_id)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_stop(port_id);
>> +
>> + return 0;
>> +}
>> +
>> +static void
>> +mempool_free_mp(struct rte_mempool *mp)
>> +{
>> + if (is_proc_primary())
>> + return rte_mempool_free(mp);
>
> As far as I remember some compilers do not like it for void.
> Just remove 'return'.
>
>> +}
>> +
>> +static int
>> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
>> +{
>> + if (is_proc_primary())
>> + return rte_eth_dev_set_mtu(port_id, mtu);
>> +
>> + return 0;
>> +}
>> +
>> /* Forward function declarations */
>> static void setup_attached_port(portid_t pi);
>> static void check_all_ports_link_status(uint32_t port_mask);
>> @@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
>> mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
>> mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
>>
>> + if (!is_proc_primary()) {
>> + rte_mp = rte_mempool_lookup(pool_name);
>> + goto err;
>
> It looks like error path, but it works in the case of success
> as well. Looks confusing.
>
>> + }
>> +
>> TESTPMD_LOG(INFO,
>> "create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
>> pool_name, nb_mbuf, mbuf_seg_size, socket_id);
>> @@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
>>
>> err:
>> if (rte_mp == NULL) {
>> - rte_exit(EXIT_FAILURE,
>> - "Creation of mbuf pool for socket %u failed: %s\n",
>> - socket_id, rte_strerror(rte_errno));
>> + if (is_proc_primary())
>> + rte_exit(EXIT_FAILURE,
>> + "Creation of mbuf pool for socket %u failed: %s\n",
>> + socket_id, rte_strerror(rte_errno));
>> + else
>> + rte_exit(EXIT_FAILURE,
>> + "Get mbuf pool for socket %u failed: %s\n",
>> + socket_id, rte_strerror(rte_errno));
>> } else if (verbose_level > 0) {
>> rte_mempool_dump(stdout, rte_mp);
>> }
>> @@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
>> uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
>> uint64_t timer_period;
>>
>> + if (num_procs > 1) {
>> + printf("multi-process not support for flushing fwd rx "
>
> rx -> Rx
> also it is better to avoid like split.
>
>> + "queues, skip the below lines and return.\n");
>> + return;
>> + }
>> +
>> /* convert to number of cycles */
>> timer_period = rte_get_timer_hz(); /* 1 second timeout */
>>
>> @@ -2511,21 +2583,24 @@ start_port(portid_t pid)
>> return -1;
>> }
>> /* configure port */
>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>> - nb_txq + nb_hairpinq,
>> - &(port->dev_conf));
>> + diag = eth_dev_configure_mp(pi,
>> + nb_rxq + nb_hairpinq,
>> + nb_txq + nb_hairpinq,
>> + &(port->dev_conf));
>> if (diag != 0) {
>> - if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> + if (rte_atomic16_cmpset(
>> + &(port->port_status),
>> + RTE_PORT_HANDLING,
>> + RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set back to stopped\n",
>
> can not -> cannot (since you touch the line anyway)
>
>> + pi);
>> printf("Fail to configure port %d\n", pi);
>> /* try to reconfigure port next time */
>> port->need_reconfig = 1;
>> return -1;
>> }
>> }
>> - if (port->need_reconfig_queues > 0) {
>> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
>> port->need_reconfig_queues = 0;
>> /* setup tx queues */
>> for (qi = 0; qi < nb_txq; qi++) {
>> @@ -2548,8 +2623,8 @@ start_port(portid_t pid)
>> if (rte_atomic16_cmpset(&(port->port_status),
>> RTE_PORT_HANDLING,
>> RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back "
>> - "to stopped\n", pi);
>> + printf("Port %d can not be set back to stopped\n",
>
> can not -> cannot
>
>> + pi);
>> printf("Fail to configure port %d tx queues\n",
>> pi);
>> /* try to reconfigure queues next time */
>> @@ -2626,16 +2701,16 @@ start_port(portid_t pid)
>> cnt_pi++;
>>
>> /* start port */
>> - diag = rte_eth_dev_start(pi);
>> + diag = eth_dev_start_mp(pi);
>> if (diag < 0) {
>> printf("Fail to start port %d: %s\n", pi,
>> rte_strerror(-diag));
>>
>> /* Fail to setup rx queue, return */
>> if (rte_atomic16_cmpset(&(port->port_status),
>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> - printf("Port %d can not be set back to "
>> - "stopped\n", pi);
>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>> + printf("Port %d can not be set back to stopped\n",
>
> can not -> cannot
>
>> + pi);
>> continue;
>> }
>>
>> @@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
>> if (port->flow_list)
>> port_flow_flush(pi);
>>
>> - if (rte_eth_dev_stop(pi) != 0)
>> + if (eth_dev_stop_mp(pi) != 0)
>> RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
>> pi);
>>
>> @@ -2834,8 +2909,10 @@ close_port(portid_t pid)
>> continue;
>> }
>>
>> - port_flow_flush(pi);
>> - rte_eth_dev_close(pi);
>> + if (is_proc_primary()) {
>> + port_flow_flush(pi);
>> + rte_eth_dev_close(pi);
>> + }
>> }
>>
>> remove_invalid_ports();
>> @@ -3101,7 +3178,7 @@ pmd_test_exit(void)
>> }
>> for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
>> if (mempools[i])
>> - rte_mempool_free(mempools[i]);
>> + mempool_free_mp(mempools[i]);
>> }
>>
>> printf("\nBye...\n");
>> @@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
>> * if unset do it here
>> */
>> if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
>> - ret = rte_eth_dev_set_mtu(portid,
>> + ret = eth_dev_set_mtu_mp(portid,
>> port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
>> if (ret)
>> printf("Failed to set MTU to %u for port %u\n",
>> @@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
>> int retval;
>> uint16_t i;
>>
>> + if (num_procs > 1) {
>> + printf("The multi-process feature doesn't support dcb.\n");
>> + return -ENOTSUP;
>> + }
>> rte_port = &ports[pid];
>>
>> memset(&port_conf, 0, sizeof(struct rte_eth_conf));
>> @@ -3787,10 +3868,6 @@ main(int argc, char** argv)
>> rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
>> rte_strerror(rte_errno));
>>
>> - if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>> - rte_exit(EXIT_FAILURE,
>> - "Secondary process type not supported.\n");
>> -
>> ret = register_eth_event_callback();
>> if (ret != 0)
>> rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
>> diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
>> index 6ca872d..3318e8f 100644
>> --- a/app/test-pmd/testpmd.h
>> +++ b/app/test-pmd/testpmd.h
>> @@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
>>
>> extern struct rte_flow_action_conntrack conntrack_context;
>>
>> +extern int proc_id;
>> +extern unsigned int num_procs;
>> +
>> +static inline bool
>> +is_proc_primary(void)
>> +{
>> + return rte_eal_process_type() == RTE_PROC_PRIMARY;
>> +}
>> +
>> static inline unsigned int
>> lcore_num(void)
>> {
>> diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
>> index b36c120..9361332 100644
>> --- a/doc/guides/rel_notes/release_21_05.rst
>> +++ b/doc/guides/rel_notes/release_21_05.rst
>> @@ -235,6 +235,7 @@ New Features
>> ``port cleanup (port_id) txq (queue_id) (free_cnt)``
>> * Added command to show link flow control info.
>> ``show port (port_id) flow_ctrl``
>> + * Added support multi-process for testpmd.
>
> Please, rebase to 21.08.
>
>>
>> * **Updated ipsec-secgw sample application.**
>>
>> diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
>> index d062165..74efa4f 100644
>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>> @@ -543,3 +543,73 @@ The command line options are:
>> bit 1 - two hairpin ports paired
>> bit 0 - two hairpin ports loop
>> The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
>> +
>> +Testpmd Multi-Process Command-line Options
>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The following are the command-line options for testpmd multi-process support:
>> +
>> +.. code-block:: console
>> +
>> + primary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
>> + --num-procs=2 --proc-id=0
>> +
>> + secondary process:
>> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
>> + --num-procs=2 --proc-id=1
>
> Do we really need --rxq=4 --txq=4 in secodary processes?
> Can't we get it from already configured device in primary
> process?
>
> Same question is applicable to --num-procs. May be testpmd
> can publish in shared memory? If yes, I'm OK to either
> do it in the patch or defer as a later improvement.
>
Yes, agree with you, it can be done for later improvement.
>> +
>> +The command line options are:
>> +
>> +* ``--num-procs=N``
>> +
>> + The number of processes which will be used.
>> +
>> +* ``--proc-id=id``
>> +
>> + The id of the current process (id < num-procs). id should be different in primary
>> + process and secondary process, which starts from '0'.
>> +
>> +Calculation rule for queue:
>> +All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
>> +Calculation rule for the testpmd to allocate queues to each process:
>> +start(queue start id) = proc_id * nb_q / num_procs；
>> +end(queue end id) = start + nb_q / num_procs；
>> +
>> +For example, if testpmd supports 4 Tx and Rx queues
>> +the 0~1 for primary process
>> +the 2~3 for secondary process
>> +
>> +The number of rings should be a multiple of the number of processes. If not,
>> +redundant queues will exist after queues are allocated to processes. After RSS is
>> +enabled, packet loss occurs when traffic is sent to all processes at the same time.
>> +Some traffic enters redundant queues and cannot be forwarded.
>> +
>> +All the dev ops is supported in primary process. While secondary process is not permitted
>> +to allocate or release shared memory, so some ops are not supported as follows::
>> +``dev_configure``
>> +``dev_start``
>> +``dev_stop``
>> +``rx_queue_setup``
>> +``tx_queue_setup``
>> +``rx_queue_release``
>> +``tx_queue_release``
>
> @Thomas, @Ferrh, shouldn't it be handled on ethdev level as
> well if it is really that strict.Yes, API modification may be handled as another patch for later improvement.
>
>> +
>> +So, any command from testpmd which calls those APIs will not be supported in secondary
>> +process, like::
>> +``port config all rxq|txq|rxd|txd <value>``
>> +``port config <port_id> rx_offload xxx on/off ``
>> +``port config <port_id> tx_offload xxx on/off``
>> +etc.
>> +
>> +Flow API is supported, it applies only on its own process on SW side, but all on HW side.
>
> Sorry, I don't understand it.
This may be confusing, I will delete the lines.
What I mean is Flow API is supported,that is it.
>
>> +
>> +Stats is supported, stats will not change when one quit and start, as they share the same
>> +buffer to store the stats. Flow rules are maintained in process level: primary and secondary
>> +has its own flow list (but one flow list in HW). The two can see all the queues, so setting
>> +the flow rules for the other is OK. But in the testpmd primary process receiving or transmitting
>> +packets from the queue allocated for secondary process is not permitted, and same for
>> +secondary process.
>> +
>> +RSS is supported, primary process and secondary process has separate queues to use, RSS
>> +will work in their own queues whether primary or secondary process.
>
> For me it sounds like secondary process has own RSS
> configuration. If so, it is false. I guess I simply
> misunderstand above paragraph.
This may be confusing, I will delete the lines.
What I mean is that primary process and secondary process has its
own queues, but both the queues are processed by RSS.
> .
>
^ permalink raw reply [flat|nested] 80+ messages in thread