From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
To: "Min Hu (Connor)" <humin29@huawei.com>,
Xiaoyun Li <xiaoyun.li@intel.com>,
Anatoly Burakov <anatoly.burakov@intel.com>
Cc: dev@dpdk.org, Lijun Ou <oulijun@huawei.com>,
Ajit Khaparde <ajit.khaparde@broadcom.com>,
Ferruh Yigit <ferruh.yigit@intel.com>
Subject: Re: [dpdk-dev] [PATCH v15] app/testpmd: support multi-process
Date: Thu, 8 Jul 2021 15:30:03 +0300 [thread overview]
Message-ID: <ca90dd4f-b23c-0e01-1c89-9b2f7734b3b5@oktetlabs.ru> (raw)
In-Reply-To: <3213dc8b-f40a-cfbe-aa69-7f8925773bfe@huawei.com>
On 7/8/21 3:20 PM, Min Hu (Connor) wrote:
> Hi, Andrew ,
>
> 在 2021/7/2 20:47, Andrew Rybchenko 写道:
>> On 7/2/21 3:09 PM, Andrew Rybchenko wrote:
>>> From: "Min Hu (Connor)" <humin29@huawei.com>
>>>
>>> For example the following commands run two testpmd processes:
>>>
>>> * the primary process:
>>>
>>> ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=0
>>>
>>> * the secondary process:
>>>
>>> ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i \
>>> --rxq=4 --txq=4 --num-procs=2 --proc-id=1
>>>
>>> Signed-off-by: Min Hu (Connor) <humin29@huawei.com>
>>> Signed-off-by: Lijun Ou <oulijun@huawei.com>
>>> Signed-off-by: Andrew Rybchenko <Andrew.Rybchenko@oktetlabs.ru>
>>> Acked-by: Xiaoyun Li <xiaoyun.li@intel.com>
>>> Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
>>> Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>
>> [snip]
>>
>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>>> index 1cdd3cdd1..a5da0c272 100644
>>> --- a/app/test-pmd/testpmd.c
>>> +++ b/app/test-pmd/testpmd.c
>>> @@ -520,6 +520,62 @@ enum rte_eth_rx_mq_mode rx_mq_mode =
>>> ETH_MQ_RX_VMDQ_DCB_RSS;
>>> */
>>> uint32_t eth_link_speed;
>>> +/*
>>> + * ID of the current process in multi-process, used to
>>> + * configure the queues to be polled.
>>> + */
>>> +int proc_id;
>>> +
>>> +/*
>>> + * Number of processes in multi-process, used to
>>> + * configure the queues to be polled.
>>> + */
>>> +unsigned int num_procs = 1;
>>> +
>>> +static int
>>> +eth_dev_configure_mp(uint16_t port_id, uint16_t nb_rx_q, uint16_t
>>> nb_tx_q,
>>> + const struct rte_eth_conf *dev_conf)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_configure(port_id, nb_rx_q, nb_tx_q,
>>> + dev_conf);
>>> + return 0;
>>> +}
>>> +
>>> +static int
>>> +eth_dev_start_mp(uint16_t port_id)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_start(port_id);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int
>>> +eth_dev_stop_mp(uint16_t port_id)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_stop(port_id);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static void
>>> +mempool_free_mp(struct rte_mempool *mp)
>>> +{
>>> + if (is_proc_primary())
>>> + rte_mempool_free(mp);
>>> +}
>>> +
>>> +static int
>>> +eth_dev_set_mtu_mp(uint16_t port_id, uint16_t mtu)
>>> +{
>>> + if (is_proc_primary())
>>> + return rte_eth_dev_set_mtu(port_id, mtu);
>>> +
>>> + return 0;
>>> +}
>>> +
>>
>> I think above functions should be removed and corresponding
>> checks should be done in caller directly since above functions
>> are used in single place only and just hide what actually
>> happens in the case of secondary process. It is very
>> misleading.
>>
> This was done as Ferruh suggested in V9, and this could reduce
> the complexity for testpmd when added by the multi-process support.
>> [snip]
>>
>>> @@ -2495,21 +2565,24 @@ start_port(portid_t pid)
>>> return -1;
>>> }
>>> /* configure port */
>>> - diag = rte_eth_dev_configure(pi, nb_rxq + nb_hairpinq,
>>> - nb_txq + nb_hairpinq,
>>> - &(port->dev_conf));
>>> + diag = eth_dev_configure_mp(pi,
>>> + nb_rxq + nb_hairpinq,
>>> + nb_txq + nb_hairpinq,
>>> + &(port->dev_conf));
>>> if (diag != 0) {
>>> - if (rte_atomic16_cmpset(&(port->port_status),
>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back "
>>> - "to stopped\n", pi);
>>> + if (rte_atomic16_cmpset(
>>> + &(port->port_status),
>>> + RTE_PORT_HANDLING,
>>> + RTE_PORT_STOPPED) == 0)
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided since
>> it just makes the review harder.This will be fixed in v16.
>>
>>> printf("Fail to configure port %d\n", pi);
>>> /* try to reconfigure port next time */
>>> port->need_reconfig = 1;
>>> return -1;
>>> }
>>> }
>>> - if (port->need_reconfig_queues > 0) {
>>> + if (port->need_reconfig_queues > 0 && is_proc_primary()) {
>>> port->need_reconfig_queues = 0;
>>> /* setup tx queues */
>>> for (qi = 0; qi < nb_txq; qi++) {
>>> @@ -2532,8 +2605,8 @@ start_port(portid_t pid)
>>> if (rte_atomic16_cmpset(&(port->port_status),
>>> RTE_PORT_HANDLING,
>>> RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back "
>>> - "to stopped\n", pi);
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided.
> This will be fixed in v16.
>>
>>> printf("Fail to configure port %d tx queues\n",
>>> pi);
>>> /* try to reconfigure queues next time */
>>> @@ -2610,16 +2683,16 @@ start_port(portid_t pid)
>>> cnt_pi++;
>>> /* start port */
>>> - diag = rte_eth_dev_start(pi);
>>> + diag = eth_dev_start_mp(pi);
>>> if (diag < 0) {
>>> printf("Fail to start port %d: %s\n", pi,
>>> rte_strerror(-diag));
>>> /* Fail to setup rx queue, return */
>>> if (rte_atomic16_cmpset(&(port->port_status),
>>> - RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> - printf("Port %d can not be set back to "
>>> - "stopped\n", pi);
>>> + RTE_PORT_HANDLING, RTE_PORT_STOPPED) == 0)
>>> + printf("Port %d cannot be set back to stopped\n",
>>> + pi);
>>
>> Unrelated changes in the patch should be avoided.
> This will be fixed in v16.
>>
>> [snip]
>>
>>> diff --git a/doc/guides/testpmd_app_ug/run_app.rst
>>> b/doc/guides/testpmd_app_ug/run_app.rst
>>> index eb4831835..348e5fcac 100644
>>> --- a/doc/guides/testpmd_app_ug/run_app.rst
>>> +++ b/doc/guides/testpmd_app_ug/run_app.rst
>>> @@ -545,3 +545,85 @@ The command line options are:
>>> bit 0 - two hairpin ports loop
>>> The default value is 0. Hairpin will use single port mode and
>>> implicit Tx flow mode.
>>> +
>>> +
>>> +Testpmd Multi-Process Command-line Options
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +The following are the command-line options for testpmd multi-process
>>> support:
>>> +
>>> +* primary process:
>>> +
>>> +.. code-block:: console
>>> +
>>> + sudo ./dpdk-testpmd --proc-type=auto -l 0-1 -- -i --rxq=4
>>> --txq=4 \
>>> + --num-procs=2 --proc-id=0
>>> +
>>> +* secondary process:
>>> +
>>> +.. code-block:: console
>>> +
>>> + sudo ./dpdk-testpmd --proc-type=auto -l 2-3 -- -i --rxq=4
>>> --txq=4 \
>>> + --num-procs=2 --proc-id=1
>>> +
>>> +The command line options are:
>>> +
>>> +* ``--num-procs=N``
>>> +
>>> + The number of processes which will be used.
>>> +
>>> +* ``--proc-id=ID``
>>> +
>>> + The ID of the current process (ID < num-procs). ID should be
>>> different in
>>> + primary process and secondary process, which starts from '0'.
>>> +
>>> +Calculation rule for queue:
>>> +All queues are allocated to different processes based on
>>> ``proc_num`` and
>>> +``proc_id``.
>>> +Calculation rule for the testpmd to allocate queues to each process:
>>> +
>>> +* start(queue start id) = proc_id * nb_q / num_procs
>>> +
>>> +* end(queue end id) = start + nb_q / num_procs
>>> +
>>> +For example, if testpmd is configured to have 4 Tx and Rx queues,
>>> +queues 0 and 1 will be used by the primary process and
>>> +queues 2 and 3 will be used by the secondary process.
>>> +
>>> +The number of queues should be a multiple of the number of
>>> processes. If not,
>>> +redundant queues will exist after queues are allocated to processes.
>>> If RSS
>>> +is enabled, packet loss occurs when traffic is sent to all processes
>>> at the same
>>> +time. Some traffic goes to redundant queues and cannot be forwarded.
>>> +
>>> +All the dev ops is supported in primary process. While secondary
>>> process is
>>> +not permitted to allocate or release shared memory, so some ops are
>>> not supported
>>> +as follows:
>>> +
>>> +- ``dev_configure``
>>> +- ``dev_start``
>>> +- ``dev_stop``
>>> +- ``rx_queue_setup``
>>> +- ``tx_queue_setup``
>>> +- ``rx_queue_release``
>>> +- ``tx_queue_release``
>>> +
>>> +So, any command from testpmd which calls those APIs will not be
>>> supported in
>>> +secondary process, like:
>>> +
>>> +.. code-block:: console
>>> +
>>> + port config all rxq|txq|rxd|txd <value>
>>> + port config <port_id> rx_offload xxx on/off
>>> + port config <port_id> tx_offload xxx on/off
>>> +
>>> +etc.
>>
>> I did the formatting cleanup, but I still think that testpmd
>> guide should not dive into such level of details. It should
>> rather highlight multi-process behaviour specifics.
>>
>> Shouldn't testpmd store state in shared memory to avoid
>> problems when primary is stopped while secondary is running
> This could be taken into consideration in future.
>
>>
>> Some testpmd features rely on reconfigure (i.e. simply change
>> configuration and set flag that reconfigure is required), but
>> configure does nothing and will simply ignore new settings.
>> So, it could look very-very confusing from user point of view.
>>
>> I'm not sure that it is acceptable to apply the patch in such
>> state and open huge number of bugs in testpmd behaviour when
>> multi-process is used.
>>
>> I'd even consider to exclude unsupported commands from help
>> etc. However, such level of care about user could be excessive
>> for test tool.
> This has been done in doc.
>>
>> IMHO, it should be no requirement to repeat the primary
>> process command-line configuration in the second process
>> command line (see --rxq=4 --txq=4 above). The information
>> should be obtained from shared state. In theory primary
>> process could even change some settings in interactive
> We think keeping the command line in consistent between primary
> and secondary is easy to understand for users. While shared memory for
> keeping in order or communicating could be performed,but this could
> be done in future patch.
>
>> mode. I think testpmd should guarantee consistent behaviour
>> even in such conditions. I.e. do not allow to stop ports
>> used by forwarding running in secondary processes.
>> Run-time queues setup and deferred start should be very
>> carefully handled as well.
> ``dev_stop`` is not allowed in secondary, which has described in doc.
I'm talking about dev_stop in primary while secondary is
running.
>>> +
>>> +Stats is supported, stats will not change when one quits and starts,
>>> as they
>>> +share the same buffer to store the stats. Flow rules are maintained
>>> in process
>>> +level: primary and secondary has its own flow list (but one flow
>>> list in HW).
>>> +The two can see all the queues, so setting the flow rules for the
>>> other is OK.
>>> +But in the testpmd primary process receiving or transmitting packets
>>> from the
>>> +queue allocated for secondary process is not permitted, and same for
>>> secondary
>>> +process.
>>> +
>>> +Flow API and RSS are supported.
>>> Thanks for your comment,
> This patch supports basic function for multi-process support in testpmd.
> I think other patches in future could enhance or optimize it, thanks.
IMHO, as I state above, current state is insufficient to
consider is a start point to be applied.
next prev parent reply other threads:[~2021-07-08 12:30 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-08 9:46 [dpdk-dev] [RFC] " Lijun Ou
2021-01-08 10:28 ` Ferruh Yigit
2021-01-09 9:54 ` oulijun
2021-01-10 12:32 ` Wisam Monther
2021-01-12 14:13 ` oulijun
2021-01-12 14:21 ` Wisam Monther
2021-01-14 2:46 ` oulijun
2021-01-20 14:06 ` [dpdk-dev] [RFC V2] " Lijun Ou
2021-03-05 1:04 ` [dpdk-dev] [PATCH] " Lijun Ou
2021-03-05 4:05 ` Ajit Khaparde
2021-03-10 11:11 ` Min Hu (Connor)
2021-03-11 2:47 ` [dpdk-dev] [PATCH v2] " Min Hu (Connor)
2021-03-22 2:27 ` Ajit Khaparde
2021-03-22 6:35 ` Min Hu (Connor)
2021-06-15 12:23 ` [dpdk-dev] [PATCH v14] " Min Hu (Connor)
2021-07-02 12:09 ` [dpdk-dev] [PATCH v15] " Andrew Rybchenko
2021-07-02 12:47 ` Andrew Rybchenko
2021-07-08 12:20 ` Min Hu (Connor)
2021-07-08 12:30 ` Andrew Rybchenko [this message]
2021-07-08 12:51 ` Min Hu (Connor)
2021-07-10 3:50 ` [dpdk-dev] [PATCH v16] " Min Hu (Connor)
2021-07-24 11:45 ` Thomas Monjalon
2021-07-26 0:26 ` Min Hu (Connor)
2021-07-26 6:30 ` Thomas Monjalon
2021-07-26 7:28 ` Min Hu (Connor)
2021-08-02 1:51 ` Min Hu (Connor)
2021-08-02 8:03 ` Thomas Monjalon
2021-08-16 18:12 ` Singh, Aman Deep
2021-08-24 12:18 ` Ferruh Yigit
2021-08-24 13:27 ` Min Hu (Connor)
2021-08-25 2:06 ` [dpdk-dev] [PATCH v17] " Min Hu (Connor)
2021-09-07 13:23 ` Ferruh Yigit
2021-09-08 0:48 ` Min Hu (Connor)
2021-03-11 9:07 ` [dpdk-dev] [PATCH v3] " Min Hu (Connor)
2021-03-20 0:58 ` Min Hu (Connor)
2021-03-22 7:07 ` [dpdk-dev] [PATCH v4] " Min Hu (Connor)
2021-03-22 11:19 ` Ferruh Yigit
2021-03-24 8:08 ` Li, Xiaoyun
2021-03-25 13:32 ` Min Hu (Connor)
2021-03-25 23:25 ` Ajit Khaparde
2021-03-26 6:46 ` Min Hu (Connor)
2021-03-25 13:17 ` [dpdk-dev] [PATCH v5] " Min Hu (Connor)
2021-03-26 6:46 ` [dpdk-dev] [PATCH v6] " Min Hu (Connor)
2021-03-26 8:52 ` [dpdk-dev] [PATCH v7] " Min Hu (Connor)
2021-03-29 7:51 ` Li, Xiaoyun
2021-03-30 1:48 ` Min Hu (Connor)
2021-03-30 1:48 ` [dpdk-dev] [PATCH v8] " Min Hu (Connor)
2021-03-30 2:17 ` Li, Xiaoyun
2021-03-30 6:36 ` Min Hu (Connor)
2021-03-30 3:11 ` Ajit Khaparde
2021-03-30 6:41 ` Min Hu (Connor)
2021-03-30 10:19 ` Ferruh Yigit
2021-03-30 10:43 ` Min Hu (Connor)
2021-04-08 10:32 ` Min Hu (Connor)
2021-04-08 13:27 ` Ferruh Yigit
2021-04-09 0:45 ` Min Hu (Connor)
2021-04-12 16:37 ` Ferruh Yigit
2021-04-15 7:54 ` Ferruh Yigit
2021-04-16 2:20 ` Min Hu (Connor)
2021-04-16 1:52 ` [dpdk-dev] [PATCH v9] " Min Hu (Connor)
2021-04-16 16:30 ` Ferruh Yigit
2021-04-17 6:12 ` Min Hu (Connor)
2021-04-17 6:12 ` [dpdk-dev] [PATCH v10] " Min Hu (Connor)
2021-04-17 22:21 ` Ferruh Yigit
2021-04-19 1:03 ` Min Hu (Connor)
2021-04-19 1:03 ` [dpdk-dev] [PATCH v11] " Min Hu (Connor)
2021-04-19 13:42 ` Ferruh Yigit
2021-04-21 9:08 ` Min Hu (Connor)
2021-04-21 8:36 ` [dpdk-dev] [PATCH v12] " Min Hu (Connor)
2021-04-22 1:18 ` [dpdk-dev] [PATCH v13] " Min Hu (Connor)
2021-06-08 8:42 ` Andrew Rybchenko
2021-06-08 10:22 ` Thomas Monjalon
2021-06-08 10:39 ` Andrew Rybchenko
2021-06-08 12:02 ` Thomas Monjalon
2021-06-08 12:36 ` Ferruh Yigit
2021-06-15 12:04 ` Min Hu (Connor)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ca90dd4f-b23c-0e01-1c89-9b2f7734b3b5@oktetlabs.ru \
--to=andrew.rybchenko@oktetlabs.ru \
--cc=ajit.khaparde@broadcom.com \
--cc=anatoly.burakov@intel.com \
--cc=dev@dpdk.org \
--cc=ferruh.yigit@intel.com \
--cc=humin29@huawei.com \
--cc=oulijun@huawei.com \
--cc=xiaoyun.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).