* [dpdk-dev] [PATCH v12] app/testpmd: support multi-process
@ 2021-04-21 9:56 Singh, Aman Deep
2021-04-22 1:18 ` Min Hu (Connor)
0 siblings, 1 reply; 3+ messages in thread
From: Singh, Aman Deep @ 2021-04-21 9:56 UTC (permalink / raw)
To: Min Hu (Connor, 'dev@dpdk.org'
Cc: Mcnamara, John, Kovacevic, Marko, Burakov, Anatoly,
andrew.rybchenko, thomas, Li, Xiaoyun, Yigit, Ferruh
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com<mailto:humin29@huawei.com>>
Signed-off-by: Lijun Ou <oulijun@huawei.com<mailto:oulijun@huawei.com>>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com<mailto:xiaoyun.li@intel.com>>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com<mailto:ajit.khaparde@broadcom.com>>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
---
v12:
* Updated doc info.
<....>
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index d062165..7414e85 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -543,3 +543,87 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
// [Minor Comment]: , not required
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
// [Minor Comment]: , not required
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs?
+end(queue end id) = start + nb_q / num_procs?
+
+For example, if testpmp supports 4 Tx and Rx queues the 0~1 for primary
// [Spelling] testpmp --> "testpmd", change Rx --> 4 Rx
+process the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If
+not, redundant queues will exist after queues are allocated to
+processes. After RSS is enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary
+process is not permitted to allocate or release shared memory, so some ops are not supported as follows::
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be
+supported in secondary process, like::
+``port config all rxq|txq|rxd|txd <value>`` ``port config <port_id>
+rx_offload xxx on/off `` ``port config <port_id> tx_offload xxx
+on/off`` etc.
+
+Flow API is supported, it applies only on its own process on SW side, but all on HW size.
// [Replace] all --> uniformly [Spelling] size --> side
+
+Stats is supported, stats will not change when one quit and start, As
+they share the same buffer to store the stats. Flow rules are
+maintained in process level: primary and secondary has its own flow
+list (but one flow list in HW). The two can see all the queues, so
+setting the flow rules for the other is OK. But in testpmd primary
// [Replace] in --> if
+process receives or transmits packets from the queue allocated for secondary process is not permitted, and same for secondary process.
// [Rephrase] is not permitted--> it is not permitted
+
+RSS is supported, primary process and secondary process has separate
+queues to use, RSS will work in their own queues whether primary or secondary process.
--
2.7.4
Tested-by: Aman Deep Singh <aman.deep.singh@intel.com<mailto:aman.deep.singh@intel.com>>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] [PATCH v12] app/testpmd: support multi-process
2021-04-21 9:56 [dpdk-dev] [PATCH v12] app/testpmd: support multi-process Singh, Aman Deep
@ 2021-04-22 1:18 ` Min Hu (Connor)
0 siblings, 0 replies; 3+ messages in thread
From: Min Hu (Connor) @ 2021-04-22 1:18 UTC (permalink / raw)
To: Singh, Aman Deep, 'dev@dpdk.org'
Cc: Mcnamara, John, Kovacevic, Marko, Burakov, Anatoly,
andrew.rybchenko, thomas, Li, Xiaoyun, Yigit, Ferruh
Hi, Singh, fixed in v13, thanks.
在 2021/4/21 17:56, Singh, Aman Deep 写道:
> This patch adds multi-process support for testpmd.
>
> The test cmd example as follows:
>
> the primary cmd:
>
> ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
>
> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>
> the secondary cmd:
>
> ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
>
> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>
> Signed-off-by: Min Hu (Connor) <humin29@huawei.com
> <mailto:humin29@huawei.com>>
>
> Signed-off-by: Lijun Ou <oulijun@huawei.com <mailto:oulijun@huawei.com>>
>
> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com <mailto:xiaoyun.li@intel.com>>
>
> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com
> <mailto:ajit.khaparde@broadcom.com>>
>
> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com
> <mailto:ferruh.yigit@intel.com>>
>
> ---
>
> v12:
>
> * Updated doc info.
>
> <….>
>
> diff --git a/doc/guides/testpmd_app_ug/run_app.rst
> b/doc/guides/testpmd_app_ug/run_app.rst
>
> index d062165..7414e85 100644
>
> --- a/doc/guides/testpmd_app_ug/run_app.rst
>
> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>
> @@ -543,3 +543,87 @@ The command line options are:
>
> bit 1 - two hairpin ports paired
>
> bit 0 - two hairpin ports loop
>
> The default value is 0. Hairpin will use single port mode and
> implicit Tx flow mode.
>
> +
>
> +Testpmd Multi-Process Command-line Options
>
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> +
>
> +The following are the command-line options for testpmd multi-process
> support:
>
> +
>
> +.. code-block:: console
>
> +
>
> + primary process:
>
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i
> --rxq=4 --txq=4 \
>
> + --num-procs=2 --proc-id=0
>
> +
>
> + secondary process:
>
> + sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i
> --rxq=4 --txq=4 \
>
> + --num-procs=2 --proc-id=1
>
> +
>
> +The command line options are:
>
> +
>
> +* ``--rxq=N``
>
> +
>
> + Set the number of Rx queues per port to N. N is the sum of queues
> used by primary
>
> + and secondary process. Primary process and secondary process should
> have separate
>
> + queues, and each should occupy at least one queue. Where N should
> be the multiple
>
> // [Minor Comment]: , not required
>
> + of number of processes.
>
> +
>
> +* ``--txq=N``
>
> +
>
> + Set the number of Tx queues per port to N. N is the sum of queues
> used by primary
>
> + and secondary process. Primary process and secondary process should
> have separate
>
> + queues, and each should occupy at least one queue. Where N should
> be the multiple
>
> // [Minor Comment]: , not required
>
> + of number of processes.
>
> +
>
> +* ``--num-procs=N``
>
> +
>
> + The number of processes which will be used.
>
> +
>
> +* ``--proc-id=id``
>
> +
>
> + The id of the current process (id < num-procs). id should be
> different in primary
>
> + process and secondary process, which starts from '0'.
>
> +
>
> +Calculation rule for queue:
>
> +All queues are allocated to different processes based on ``proc_num``
> and ``proc_id``.
>
> +Calculation rule for the testpmd to allocate queues to each process:
>
> +start(queue start id) = proc_id * nb_q / num_procs?
>
> +end(queue end id) = start + nb_q / num_procs?
>
> +
>
> +For example, if testpmp supports 4 Tx and Rx queues the 0~1 for primary
>
> // [Spelling] testpmp à “testpmd”, change Rx à 4 Rx
>
> +process the 2~3 for secondary process
>
> +
>
> +The number of rings should be a multiple of the number of processes. If
>
> +not, redundant queues will exist after queues are allocated to
>
> +processes. After RSS is enabled, packet loss occurs when traffic is
> sent to all processes at the same time.
>
> +Some traffic enters redundant queues and cannot be forwarded.
>
> +
>
> +All the dev ops is supported in primary process. While secondary
>
> +process is not permitted to allocate or release shared memory, so some
> ops are not supported as follows::
>
> +``dev_configure``
>
> +``dev_start``
>
> +``dev_stop``
>
> +``rx_queue_setup``
>
> +``tx_queue_setup``
>
> +``rx_queue_release``
>
> +``tx_queue_release``
>
> +
>
> +So, any command from testpmd which calls those APIs will not be
>
> +supported in secondary process, like::
>
> +``port config all rxq|txq|rxd|txd <value>`` ``port config <port_id>
>
> +rx_offload xxx on/off `` ``port config <port_id> tx_offload xxx
>
> +on/off`` etc.
>
> +
>
> +Flow API is supported, it applies only on its own process on SW side,
> but all on HW size.
>
> // [Replace] all à uniformly [Spelling] size --> side
>
> +
>
> +Stats is supported, stats will not change when one quit and start, As
>
> +they share the same buffer to store the stats. Flow rules are
>
> +maintained in process level: primary and secondary has its own flow
>
> +list (but one flow list in HW). The two can see all the queues, so
>
> +setting the flow rules for the other is OK. But in testpmd primary
>
> // [Replace] in à if
>
> +process receives or transmits packets from the queue allocated for
> secondary process is not permitted, and same for secondary process.
>
> // [Rephrase] is not permittedà it is not permitted
>
> +
>
> +RSS is supported, primary process and secondary process has separate
>
> +queues to use, RSS will work in their own queues whether primary or
> secondary process.
>
> --
>
> 2.7.4
>
> Tested-by: Aman Deep Singh <aman.deep.singh@intel.com
> <mailto:aman.deep.singh@intel.com>>
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* [dpdk-dev] [PATCH] app/testpmd: support multi-process
@ 2021-03-05 1:04 Lijun Ou
2021-04-21 8:36 ` [dpdk-dev] [PATCH v12] " Min Hu (Connor)
0 siblings, 1 reply; 3+ messages in thread
From: Lijun Ou @ 2021-03-05 1:04 UTC (permalink / raw)
To: ferruh.yigit; +Cc: dev, linuxarm
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./testpmd -w xxx --file-prefix=xx -l 0-1 -n 2 -- -i\
--rxq=16 --txq=16 --num-procs=2 --proc-id=0
the secondary cmd:
./testpmd -w xxx --file-prefix=xx -l 2-3 -n 2 -- -i\
--rxq=16 --txq=16 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
---
app/test-pmd/cmdline.c | 12 +++-
app/test-pmd/config.c | 9 ++-
app/test-pmd/parameters.c | 11 ++++
app/test-pmd/testpmd.c | 138 ++++++++++++++++++++++++++++++----------------
app/test-pmd/testpmd.h | 7 +++
5 files changed, 127 insertions(+), 50 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index 8b0f7d5..610e850 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -72,8 +72,6 @@
#include "cmdline_tm.h"
#include "bpf_cmd.h"
-static struct cmdline *testpmd_cl;
-
static void cmd_reconfig_device_queue(portid_t id, uint8_t dev, uint8_t queue);
/* *** Help command with introduction. *** */
@@ -5353,6 +5351,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
@@ -17316,6 +17320,10 @@ prompt(void)
printf("Cannot set exit function for cmdline\n");
cmdline_interact(testpmd_cl);
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, testpmd_cl->s_in);
+ close(testpmd_fd_copy);
+ }
if (ret != 0)
cmdline_stdin_exit(testpmd_cl);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index 2435c26..8ab8153 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2827,6 +2827,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2844,7 +2846,10 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2861,6 +2866,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index c8acd5d..1a5bf25 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -45,6 +45,8 @@
#include <rte_flow.h>
#include "testpmd.h"
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
static void
usage(char* progname)
@@ -605,6 +607,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1366,6 +1370,13 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 8) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index 1a57324..bbd45d9 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -63,6 +63,8 @@
#include "testpmd.h"
+int testpmd_fd_copy = 500; /* the copy of STDIN_FILENO */
+
#ifndef MAP_HUGETLB
/* FreeBSD may not have MAP_HUGETLB (in fact, it probably doesn't) */
#define HUGE_FLAG (0x40000)
@@ -125,6 +127,9 @@ uint8_t port_numa[RTE_MAX_ETHPORTS];
*/
uint8_t rxring_numa[RTE_MAX_ETHPORTS];
+int proc_id;
+unsigned int num_procs = 1;
+
/*
* Store specified sockets on which TX ring to be used by ports
* is allocated.
@@ -982,16 +987,26 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
/* wrapper to rte_mempool_create() */
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size, socket_id);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, socket_id);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_ANON:
{
- rte_mp = rte_mempool_create_empty(pool_name, nb_mbuf,
- mb_size, (unsigned int) mb_mempool_cache,
- sizeof(struct rte_pktmbuf_pool_private),
- socket_id, mempool_flags);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_mempool_create_empty(pool_name,
+ nb_mbuf, mb_size,
+ (unsigned int)mb_mempool_cache,
+ sizeof(struct rte_pktmbuf_pool_private),
+ socket_id, mempool_flags);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
if (rte_mp == NULL)
goto err;
@@ -1021,9 +1036,13 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
TESTPMD_LOG(INFO, "preferred mempool ops selected: %s\n",
rte_mbuf_best_mempool_ops());
- rte_mp = rte_pktmbuf_pool_create(pool_name, nb_mbuf,
- mb_mempool_cache, 0, mbuf_seg_size,
- heap_socket);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_mp = rte_pktmbuf_pool_create(pool_name,
+ nb_mbuf, mb_mempool_cache, 0,
+ mbuf_seg_size, heap_socket);
+ else
+ rte_mp = rte_mempool_lookup(pool_name);
+
break;
}
case MP_ALLOC_XBUF:
@@ -1994,6 +2013,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2503,21 +2528,28 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_configure(pi,
+ nb_rxq + nb_hairpinq,
nb_txq + nb_hairpinq,
&(port->dev_conf));
- if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
- printf("Fail to configure port %d\n", pi);
- /* try to reconfigure port next time */
- port->need_reconfig = 1;
- return -1;
+ if (diag != 0) {
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set "
+ "back to stopped\n", pi);
+ printf("Fail to configure port %d\n",
+ pi);
+ /* try to reconfigure port next time */
+ port->need_reconfig = 1;
+ return -1;
+ }
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 &&
+ rte_eal_process_type() == RTE_PROC_PRIMARY) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2618,15 +2650,18 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- if (rte_eth_dev_start(pi) < 0) {
- printf("Fail to start port %d\n", pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+ diag = rte_eth_dev_start(pi);
+ if (diag < 0) {
+ printf("Fail to start port %d\n", pi);
- /* Fail to setup rx queue, return */
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
+ /* Fail to setup rx queue, return */
+ if (rte_atomic16_cmpset(&(port->port_status),
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to "
"stopped\n", pi);
- continue;
+ continue;
+ }
}
if (rte_atomic16_cmpset(&(port->port_status),
@@ -2755,7 +2790,8 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY &&
+ rte_eth_dev_stop(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2824,8 +2860,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ port_flow_flush(pi);
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+ rte_eth_dev_close(pi);
}
remove_invalid_ports();
@@ -3089,7 +3127,7 @@ pmd_test_exit(void)
}
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
- if (mempools[i])
+ if (rte_eal_process_type() == RTE_PROC_PRIMARY && mempools[i])
rte_mempool_free(mempools[i]);
}
@@ -3611,6 +3649,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3709,13 +3751,6 @@ init_port(void)
}
static void
-force_quit(void)
-{
- pmd_test_exit();
- prompt_exit();
-}
-
-static void
print_stats(void)
{
uint8_t i;
@@ -3746,12 +3781,16 @@ signal_handler(int signum)
if (latencystats_enabled != 0)
rte_latencystats_uninit();
#endif
- force_quit();
/* Set flag to indicate the force termination. */
f_quit = 1;
- /* exit with the expected status */
- signal(signum, SIG_DFL);
- kill(getpid(), signum);
+ if (interactive == 1) {
+ dup2(testpmd_cl->s_in, testpmd_fd_copy);
+ close(testpmd_cl->s_in);
+ } else {
+ dup2(0, testpmd_fd_copy);
+ close(0);
+ }
+
}
}
@@ -3776,10 +3815,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
@@ -3875,8 +3910,10 @@ main(int argc, char** argv)
}
}
- if (!no_device_start && start_port(RTE_PORT_ALL) != 0)
+ if (!no_device_start && start_port(RTE_PORT_ALL) != 0) {
+ pmd_test_exit();
rte_exit(EXIT_FAILURE, "Start ports failed\n");
+ }
/* set all ports to promiscuous mode by default */
RTE_ETH_FOREACH_DEV(port_id) {
@@ -3922,6 +3959,8 @@ main(int argc, char** argv)
}
prompt();
pmd_test_exit();
+ if (unlikely(f_quit == 1))
+ prompt_exit();
} else
#endif
{
@@ -3957,6 +3996,11 @@ main(int argc, char** argv)
printf("Press enter to exit\n");
rc = read(0, &c, 1);
pmd_test_exit();
+ if (unlikely(f_quit == 1)) {
+ dup2(testpmd_fd_copy, 0);
+ close(testpmd_fd_copy);
+ prompt_exit();
+ }
if (rc < 0)
return 1;
}
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index ce83f31..6d13417 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -13,6 +13,7 @@
#include <rte_gso.h>
#include <cmdline.h>
#include <sys/queue.h>
+#include "cmdline.h"
#define RTE_PORT_ALL (~(portid_t)0x0)
@@ -24,6 +25,10 @@
#define RTE_PORT_CLOSED (uint16_t)2
#define RTE_PORT_HANDLING (uint16_t)3
+uint8_t f_quit;
+int testpmd_fd_copy;
+struct cmdline *testpmd_cl;
+
/*
* It is used to allocate the memory for hash key.
* The hash key size is NIC dependent.
@@ -422,6 +427,8 @@ extern uint64_t noisy_lkup_mem_sz;
extern uint64_t noisy_lkup_num_writes;
extern uint64_t noisy_lkup_num_reads;
extern uint64_t noisy_lkup_num_reads_writes;
+extern int proc_id;
+extern unsigned int num_procs;
extern uint8_t dcb_config;
extern uint8_t dcb_test;
--
2.7.4
^ permalink raw reply [flat|nested] 3+ messages in thread
* [dpdk-dev] [PATCH v12] app/testpmd: support multi-process
2021-03-05 1:04 [dpdk-dev] [PATCH] " Lijun Ou
@ 2021-04-21 8:36 ` Min Hu (Connor)
0 siblings, 0 replies; 3+ messages in thread
From: Min Hu (Connor) @ 2021-04-21 8:36 UTC (permalink / raw)
To: dev
Cc: ferruh.yigit, john.mcnamara, marko.kovacevic, anatoly.burakov,
andrew.rybchenko, thomas, xiaoyun.li
This patch adds multi-process support for testpmd.
The test cmd example as follows:
the primary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=0
the secondary cmd:
./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i \
--rxq=4 --txq=4 --num-procs=2 --proc-id=1
Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
Signed-off-by: Lijun Ou <oulijun@huawei.com>
Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
v12:
* Updated doc info.
v11:
* Fixed some minor syntax.
v10:
* Hid process type checks behind new functions.
* Added comments.
v9:
* Updated release notes and rst doc.
* Deleted deprecated codes.
* move macro and variable.
v8:
* Added warning info about queue numbers and process numbers.
v7:
* Fixed compiling error for unexpected unindent.
v6:
* Add rte flow description for multiple process.
v5:
* Fixed run_app.rst for multiple process description.
* Fix compiling error.
v4:
* Fixed minimum vlaue of Rxq or Txq in doc.
v3:
* Fixed compiling error using gcc10.0.
v2:
* Added document for this patch.
---
app/test-pmd/cmdline.c | 6 ++
app/test-pmd/config.c | 21 +++++-
app/test-pmd/parameters.c | 11 +++
app/test-pmd/testpmd.c | 129 ++++++++++++++++++++++++++-------
app/test-pmd/testpmd.h | 9 +++
doc/guides/rel_notes/release_21_05.rst | 1 +
doc/guides/testpmd_app_ug/run_app.rst | 84 +++++++++++++++++++++
7 files changed, 234 insertions(+), 27 deletions(-)
diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c
index ccaeefa..b6bf4cd 100644
--- a/app/test-pmd/cmdline.c
+++ b/app/test-pmd/cmdline.c
@@ -5357,6 +5357,12 @@ cmd_set_flush_rx_parsed(void *parsed_result,
__rte_unused void *data)
{
struct cmd_set_flush_rx *res = parsed_result;
+
+ if (num_procs > 1 && (strcmp(res->mode, "on") == 0)) {
+ printf("multi-process doesn't support to flush rx queues.\n");
+ return;
+ }
+
no_flush_rx = (uint8_t)((strcmp(res->mode, "on") == 0) ? 0 : 1);
}
diff --git a/app/test-pmd/config.c b/app/test-pmd/config.c
index ccb9bd3..3c7a378 100644
--- a/app/test-pmd/config.c
+++ b/app/test-pmd/config.c
@@ -2924,6 +2924,8 @@ rss_fwd_config_setup(void)
queueid_t rxq;
queueid_t nb_q;
streamid_t sm_id;
+ int start;
+ int end;
nb_q = nb_rxq;
if (nb_q > nb_txq)
@@ -2941,7 +2943,22 @@ rss_fwd_config_setup(void)
init_fwd_streams();
setup_fwd_config_of_each_lcore(&cur_fwd_config);
- rxp = 0; rxq = 0;
+
+ if (proc_id > 0 && nb_q % num_procs)
+ printf("Warning! queue numbers should be multiple of "
+ "processes, or packet loss will happen.\n");
+
+ /**
+ * In multi-process, All queues are allocated to different
+ * processes based on num_procs and proc_id. For example:
+ * if supports 4 queues(nb_q), 2 processes(num_procs),
+ * the 0~1 queue for primary process.
+ * the 2~3 queue for secondary process.
+ */
+ start = proc_id * nb_q / num_procs;
+ end = start + nb_q / num_procs;
+ rxp = 0;
+ rxq = start;
for (sm_id = 0; sm_id < cur_fwd_config.nb_fwd_streams; sm_id++) {
struct fwd_stream *fs;
@@ -2958,6 +2975,8 @@ rss_fwd_config_setup(void)
continue;
rxp = 0;
rxq++;
+ if (rxq >= end)
+ rxq = start;
}
}
diff --git a/app/test-pmd/parameters.c b/app/test-pmd/parameters.c
index f3954c1..ece05c1 100644
--- a/app/test-pmd/parameters.c
+++ b/app/test-pmd/parameters.c
@@ -508,6 +508,9 @@ parse_link_speed(int n)
void
launch_args_parse(int argc, char** argv)
{
+#define PARAM_PROC_ID "proc-id"
+#define PARAM_NUM_PROCS "num-procs"
+
int n, opt;
char **argvopt;
int opt_idx;
@@ -625,6 +628,8 @@ launch_args_parse(int argc, char** argv)
{ "rx-mq-mode", 1, 0, 0 },
{ "record-core-cycles", 0, 0, 0 },
{ "record-burst-stats", 0, 0, 0 },
+ { PARAM_NUM_PROCS, 1, 0, 0 },
+ { PARAM_PROC_ID, 1, 0, 0 },
{ 0, 0, 0, 0 },
};
@@ -1391,6 +1396,12 @@ launch_args_parse(int argc, char** argv)
record_core_cycles = 1;
if (!strcmp(lgopts[opt_idx].name, "record-burst-stats"))
record_burst_stats = 1;
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_NUM_PROCS, 9) == 0)
+ num_procs = atoi(optarg);
+ if (strncmp(lgopts[opt_idx].name,
+ PARAM_PROC_ID, 7) == 0)
+ proc_id = atoi(optarg);
break;
case 'h':
usage(argv[0]);
diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
index d4be23f..afa2a6b 100644
--- a/app/test-pmd/testpmd.c
+++ b/app/test-pmd/testpmd.c
@@ -518,6 +518,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode = ETH_MQ_RX_VMDQ_DCB_RSS;
*/
uint32_t eth_link_speed;
+/*
+ * Id of the current process in multi-process, used to
+ * configure the queues to be polled.
+ */
+int proc_id;
+
+/*
+ * Number of processes in multi-process, used to
+ * configure the queues to be polled.
+ */
+unsigned int num_procs = 1;
+
+static int
+eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t nb_tx_q,
+ const struct rte_eth_conf *dev_conf)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
+ dev_conf);
+ return 0;
+}
+
+static int
+eth_dev_start_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_start(port_id);
+
+ return 0;
+}
+
+static int
+eth_dev_stop_mp(uint16_t port_id)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_stop(port_id);
+
+ return 0;
+}
+
+static void
+mempool_free_mp(struct rte_mempool *mp)
+{
+ if (is_proc_primary())
+ return rte_mempool_free(mp);
+}
+
+static int
+eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
+{
+ if (is_proc_primary())
+ return rte_eth_dev_set_mtu(port_id, mtu);
+
+ return 0;
+}
+
/* Forward function declarations */
static void setup_attached_port(portid_t pi);
static void check_all_ports_link_status(uint32_t port_mask);
@@ -977,6 +1033,11 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
mb_size = sizeof(struct rte_mbuf) + mbuf_seg_size;
mbuf_poolname_build(socket_id, pool_name, sizeof(pool_name), size_idx);
+ if (!is_proc_primary()) {
+ rte_mp = rte_mempool_lookup(pool_name);
+ goto err;
+ }
+
TESTPMD_LOG(INFO,
"create a new mbuf pool <%s>: n=%u, size=%u, socket=%u\n",
pool_name, nb_mbuf, mbuf_seg_size, socket_id);
@@ -1059,9 +1120,14 @@ mbuf_pool_create(uint16_t mbuf_seg_size, unsigned nb_mbuf,
err:
if (rte_mp == NULL) {
- rte_exit(EXIT_FAILURE,
- "Creation of mbuf pool for socket %u failed: %s\n",
- socket_id, rte_strerror(rte_errno));
+ if (is_proc_primary())
+ rte_exit(EXIT_FAILURE,
+ "Creation of mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
+ else
+ rte_exit(EXIT_FAILURE,
+ "Get mbuf pool for socket %u failed: %s\n",
+ socket_id, rte_strerror(rte_errno));
} else if (verbose_level > 0) {
rte_mempool_dump(stdout, rte_mp);
}
@@ -2002,6 +2068,12 @@ flush_fwd_rx_queues(void)
uint64_t prev_tsc = 0, diff_tsc, cur_tsc, timer_tsc = 0;
uint64_t timer_period;
+ if (num_procs > 1) {
+ printf("multi-process not support for flushing fwd rx "
+ "queues, skip the below lines and return.\n");
+ return;
+ }
+
/* convert to number of cycles */
timer_period = rte_get_timer_hz(); /* 1 second timeout */
@@ -2511,21 +2583,24 @@ start_port(portid_t pid)
return -1;
}
/* configure port */
- diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
- nb_txq + nb_hairpinq,
- &(port->dev_conf));
+ diag = eth_dev_configure_mp(pi,
+ nb_rxq + nb_hairpinq,
+ nb_txq + nb_hairpinq,
+ &(port->dev_conf));
if (diag != 0) {
- if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ if (rte_atomic16_cmpset(
+ &(port->port_status),
+ RTE_PORT_HANDLING,
+ RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d\n", pi);
/* try to reconfigure port next time */
port->need_reconfig = 1;
return -1;
}
}
- if (port->need_reconfig_queues > 0) {
+ if (port->need_reconfig_queues > 0 && is_proc_primary()) {
port->need_reconfig_queues = 0;
/* setup tx queues */
for (qi = 0; qi < nb_txq; qi++) {
@@ -2548,8 +2623,8 @@ start_port(portid_t pid)
if (rte_atomic16_cmpset(&(port->port_status),
RTE_PORT_HANDLING,
RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back "
- "to stopped\n", pi);
+ printf("Port %d can not be set back to stopped\n",
+ pi);
printf("Fail to configure port %d tx queues\n",
pi);
/* try to reconfigure queues next time */
@@ -2626,16 +2701,16 @@ start_port(portid_t pid)
cnt_pi++;
/* start port */
- diag = rte_eth_dev_start(pi);
+ diag = eth_dev_start_mp(pi);
if (diag < 0) {
printf("Fail to start port %d: %s\n", pi,
rte_strerror(-diag));
/* Fail to setup rx queue, return */
if (rte_atomic16_cmpset(&(port->port_status),
- RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
- printf("Port %d can not be set back to "
- "stopped\n", pi);
+ RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
+ printf("Port %d can not be set back to stopped\n",
+ pi);
continue;
}
@@ -2765,7 +2840,7 @@ stop_port(portid_t pid)
if (port->flow_list)
port_flow_flush(pi);
- if (rte_eth_dev_stop(pi) != 0)
+ if (eth_dev_stop_mp(pi) != 0)
RTE_LOG(ERR, EAL, "rte_eth_dev_stop failed for port %u\n",
pi);
@@ -2834,8 +2909,10 @@ close_port(portid_t pid)
continue;
}
- port_flow_flush(pi);
- rte_eth_dev_close(pi);
+ if (is_proc_primary()) {
+ port_flow_flush(pi);
+ rte_eth_dev_close(pi);
+ }
}
remove_invalid_ports();
@@ -3101,7 +3178,7 @@ pmd_test_exit(void)
}
for (i = 0 ; i < RTE_DIM(mempools) ; i++) {
if (mempools[i])
- rte_mempool_free(mempools[i]);
+ mempool_free_mp(mempools[i]);
}
printf("\nBye...\n");
@@ -3432,7 +3509,7 @@ update_jumbo_frame_offload(portid_t portid)
* if unset do it here
*/
if ((rx_offloads & DEV_RX_OFFLOAD_JUMBO_FRAME) == 0) {
- ret = rte_eth_dev_set_mtu(portid,
+ ret = eth_dev_set_mtu_mp(portid,
port->dev_conf.rxmode.max_rx_pkt_len - eth_overhead);
if (ret)
printf("Failed to set MTU to %u for port %u\n",
@@ -3622,6 +3699,10 @@ init_port_dcb_config(portid_t pid,
int retval;
uint16_t i;
+ if (num_procs > 1) {
+ printf("The multi-process feature doesn't support dcb.\n");
+ return -ENOTSUP;
+ }
rte_port = &ports[pid];
memset(&port_conf, 0, sizeof(struct rte_eth_conf));
@@ -3787,10 +3868,6 @@ main(int argc, char** argv)
rte_exit(EXIT_FAILURE, "Cannot init EAL: %s\n",
rte_strerror(rte_errno));
- if (rte_eal_process_type() == RTE_PROC_SECONDARY)
- rte_exit(EXIT_FAILURE,
- "Secondary process type not supported.\n");
-
ret = register_eth_event_callback();
if (ret != 0)
rte_exit(EXIT_FAILURE, "Cannot register for ethdev events");
diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h
index 9530ec5..e2939ea 100644
--- a/app/test-pmd/testpmd.h
+++ b/app/test-pmd/testpmd.h
@@ -632,6 +632,15 @@ extern enum rte_eth_rx_mq_mode rx_mq_mode;
extern struct rte_flow_action_conntrack conntrack_context;
+extern int proc_id;
+extern unsigned int num_procs;
+
+static inline bool
+is_proc_primary(void)
+{
+ return rte_eal_process_type() == RTE_PROC_PRIMARY;
+}
+
static inline unsigned int
lcore_num(void)
{
diff --git a/doc/guides/rel_notes/release_21_05.rst b/doc/guides/rel_notes/release_21_05.rst
index 668fca8..cc88aea 100644
--- a/doc/guides/rel_notes/release_21_05.rst
+++ b/doc/guides/rel_notes/release_21_05.rst
@@ -230,6 +230,7 @@ New Features
* Added commands to construct conntrack context and relevant indirect
action handle creation, update for conntrack action as well as conntrack
item matching.
+ * Added support multi-process for testpmd.
* **Updated ipsec-secgw sample application.**
diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst
index d062165..7414e85 100644
--- a/doc/guides/testpmd_app_ug/run_app.rst
+++ b/doc/guides/testpmd_app_ug/run_app.rst
@@ -543,3 +543,87 @@ The command line options are:
bit 1 - two hairpin ports paired
bit 0 - two hairpin ports loop
The default value is 0. Hairpin will use single port mode and implicit Tx flow mode.
+
+Testpmd Multi-Process Command-line Options
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The following are the command-line options for testpmd multi-process support:
+
+.. code-block:: console
+
+ primary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 0-1 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=0
+
+ secondary process:
+ sudo ./dpdk-testpmd -a xxx --proc-type=auto -l 2-3 -- -i --rxq=4 --txq=4 \
+ --num-procs=2 --proc-id=1
+
+The command line options are:
+
+* ``--rxq=N``
+
+ Set the number of Rx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--txq=N``
+
+ Set the number of Tx queues per port to N. N is the sum of queues used by primary
+ and secondary process. Primary process and secondary process should have separate
+ queues, and each should occupy at least one queue. Where N should be the multiple
+ of number of processes.
+
+* ``--num-procs=N``
+
+ The number of processes which will be used.
+
+* ``--proc-id=id``
+
+ The id of the current process (id < num-procs). id should be different in primary
+ process and secondary process, which starts from '0'.
+
+Calculation rule for queue:
+All queues are allocated to different processes based on ``proc_num`` and ``proc_id``.
+Calculation rule for the testpmd to allocate queues to each process:
+start(queue start id) = proc_id * nb_q / num_procs;
+end(queue end id) = start + nb_q / num_procs;
+
+For example, if testpmp supports 4 Tx and Rx queues
+the 0~1 for primary process
+the 2~3 for secondary process
+
+The number of rings should be a multiple of the number of processes. If not,
+redundant queues will exist after queues are allocated to processes. After RSS is
+enabled, packet loss occurs when traffic is sent to all processes at the same time.
+Some traffic enters redundant queues and cannot be forwarded.
+
+All the dev ops is supported in primary process. While secondary process is not permitted
+to allocate or release shared memory, so some ops are not supported as follows::
+``dev_configure``
+``dev_start``
+``dev_stop``
+``rx_queue_setup``
+``tx_queue_setup``
+``rx_queue_release``
+``tx_queue_release``
+
+So, any command from testpmd which calls those APIs will not be supported in secondary
+process, like::
+``port config all rxq|txq|rxd|txd <value>``
+``port config <port_id> rx_offload xxx on/off ``
+``port config <port_id> tx_offload xxx on/off``
+etc.
+
+Flow API is supported, it applies only on its own process on SW side, but all on HW size.
+
+Stats is supported, stats will not change when one quit and start, As they share the same
+buffer to store the stats. Flow rules are maintained in process level: primary and secondary
+has its own flow list (but one flow list in HW). The two can see all the queues, so setting
+the flow rules for the other is OK. But in testpmd primary process receives or transmits
+packets from the queue allocated for secondary process is not permitted, and same for
+secondary process.
+
+RSS is supported, primary process and secondary process has separate queues to use, RSS
+will work in their own queues whether primary or secondary process.
--
2.7.4
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-04-22 1:18 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-21 9:56 [dpdk-dev] [PATCH v12] app/testpmd: support multi-process Singh, Aman Deep
2021-04-22 1:18 ` Min Hu (Connor)
-- strict thread matches above, loose matches on Subject: below --
2021-03-05 1:04 [dpdk-dev] [PATCH] " Lijun Ou
2021-04-21 8:36 ` [dpdk-dev] [PATCH v12] " Min Hu (Connor)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).