Re: [PATCH v3] app/testpmd: fix secondary process not forwarding

[Ferruh Yigit <ferruh.yigit@amd.com>]

On 3/7/2023 3:25 AM, He, ShiyangX wrote:
> 
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@amd.com>
>> Sent: Monday, March 6, 2023 11:06 PM
>> To: He, ShiyangX <shiyangx.he@intel.com>; dev@dpdk.org
>> Cc: Zhou, YidingX <yidingx.zhou@intel.com>; stable@dpdk.org; Zhang, Yuying
>> <yuying.zhang@intel.com>; Singh, Aman Deep
>> <aman.deep.singh@intel.com>; Burakov, Anatoly
>> <anatoly.burakov@intel.com>; Matan Azrad <matan@nvidia.com>; Dmitry
>> Kozlyuk <dmitry.kozliuk@gmail.com>
>> Subject: Re: [PATCH v3] app/testpmd: fix secondary process not forwarding
>>
>> On 2/23/2023 2:41 PM, Shiyang He wrote:
>>> Under multi-process scenario, the secondary process gets queue state
>>> from the wrong location (the global variable 'ports'). Therefore, the
>>> secondary process can not forward since "stream_init" is not called.
>>>
>>> This commit fixes the issue by calling 'rte_eth_rx/tx_queue_info_get'
>>> to get queue state from shared memory.
>>>
>>> Fixes: 3c4426db54fc ("app/testpmd: do not poll stopped queues")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Shiyang He <shiyangx.he@intel.com>
>>> Acked-by: Yuying Zhang <yuying.zhang@intel.com>
>>>
>>> v3: Add return value description
>>> ---
>>>  app/test-pmd/testpmd.c | 45
>>> ++++++++++++++++++++++++++++++++++++++++--
>>>  1 file changed, 43 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index
>>> 0c14325b8d..a050472aea 100644
>>> --- a/app/test-pmd/testpmd.c
>>> +++ b/app/test-pmd/testpmd.c
>>> @@ -2418,9 +2418,50 @@ start_packet_forwarding(int with_tx_first)
>>>  	if (!pkt_fwd_shared_rxq_check())
>>>  		return;
>>>
>>> -	if (stream_init != NULL)
>>> -		for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++)
>>> +	if (stream_init != NULL) {
>>> +		for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++) {
>>> +			if (rte_eal_process_type() == RTE_PROC_SECONDARY)
>> {
>>> +				struct fwd_stream *fs = fwd_streams[i];
>>> +				struct rte_eth_rxq_info rx_qinfo;
>>> +				struct rte_eth_txq_info tx_qinfo;
>>> +				int32_t rc;
>>> +				rc = rte_eth_rx_queue_info_get(fs->rx_port,
>>> +						fs->rx_queue, &rx_qinfo);
>>> +				if (rc == 0) {
>>> +					ports[fs->rx_port].rxq[fs-
>>> rx_queue].state =
>>> +						rx_qinfo.queue_state;
>>> +				} else if (rc == -ENOTSUP) {
>>> +					/* Set the rxq state to
>> RTE_ETH_QUEUE_STATE_STARTED
>>> +					 * to ensure that the PMDs do not
>> implement
>>> +					 * rte_eth_rx_queue_info_get can
>> forward.
>>> +					 */
>>> +					ports[fs->rx_port].rxq[fs-
>>> rx_queue].state =
>>> +
>> 	RTE_ETH_QUEUE_STATE_STARTED;
>>> +				} else {
>>> +					TESTPMD_LOG(WARNING,
>>> +						"Failed to get rx queue
>> info\n");
>>> +				}
>>> +
>>> +				rc = rte_eth_tx_queue_info_get(fs->tx_port,
>>> +						fs->tx_queue, &tx_qinfo);
>>> +				if (rc == 0) {
>>> +					ports[fs->tx_port].txq[fs-
>>> tx_queue].state =
>>> +						tx_qinfo.queue_state;
>>> +				} else if (rc == -ENOTSUP) {
>>> +					/* Set the txq state to
>> RTE_ETH_QUEUE_STATE_STARTED
>>> +					 * to ensure that the PMDs do not
>> implement
>>> +					 * rte_eth_tx_queue_info_get can
>> forward.
>>> +					 */
>>> +					ports[fs->tx_port].txq[fs-
>>> tx_queue].state =
>>> +
>> 	RTE_ETH_QUEUE_STATE_STARTED;
>>> +				} else {
>>> +					TESTPMD_LOG(WARNING,
>>> +						"Failed to get tx queue
>> info\n");
>>> +				}
>>> +			}
>>>  			stream_init(fwd_streams[i]);
>>> +		}
>>> +	}
>>>
>>
>>
>> Testpmd duplicates some dpdk/ethdev state/config in application level, and
>> this can bite in multiple cases, as it is happening here.
>>
>> I am not sure if this was a design decision, but I think instead of testpmd
>> storing ethdev related state/config in application level, it should store only
>> application level state/config, and when ethdev related state/config is
>> required app should get it directly from ethdev.
>>
>> It may be too late already for testpmd, there is a mixed usage, but I am for
>> preferring this approach when there is an opportunity.
>>
>>
>>
>> For above issue, why queue state needs to be stored in application level 'port'
>> variable?
>> Where is this queue state used?
>>
>> Can it work to get queue state directly from ethdev where this state is used,
>> instead of storing it in the 'port' variable in advance?
>>
>> And perhaps testpmd 'port' variable can be updated there, both for primary
>> and secondary, for backward compatibility (other existing users of this queue
>> state).
>>
>> What do you think?
> 
> Thanks for your comments!
> 
> It is an effective method to get queue state directly from ethdev where this state is used.
> I also don't know the design meaning of the 'ports' variable. If modification is needed,
> a higher level of design and more work are required.
> 
> As a bug fix, apart from extracting the code block into a function, is the solution feasible?

Hi Shiyang,

As a bug fix, this issue (testpmd stored state not being up to date for
secondary process) looks like have potential to occur many different
flavors, that is why what about having a central update?

I think 'start_port()' can be a good place for this kind of update:

start_port() {

	...
	if (secondary)
		update_state()
}

update_state() {
	update_queue_state()
}

update_queue_state() {
	<your code goes here>
}


Having secondary checks and updates in multiple places can make code
harder to understand.

What do you think to update as above?