DPDK22 issue: Unable to set more than 4 queues in Azure

DPDK22 issue: Unable to set more than 4 queues in Azure

[Nageswara Rao]

[-- Attachment #1: Type: text/plain, Size: 3102 bytes --]

Hi All,

We are observing the following issue with DPDK22.11. We didn’t find any
upstream patches for this issue on the DPDK github. Is there any known
issue, please let us know.



*Issue:*

On Azure platform, we are unable to configure more than 4 queues. When we
try to configure more than 4 queues its failing with “EAL: Cannot send more
than 8 FDs” error.

Here I am pasting the working and failing testpmd logs.

Please note that this issue is not observed in DPDK 21.11.



*DPDK Testpmd*:
*Working case with 4 rx & 4 tx queues:*
./build/app/dpdk-testpmd -l 0-3 -a 38d9:00:02.0 -- --nb-cores=3 --rxq=4
--txq=4 --burst=64 --mbcache=512 --max-pkt-len=128 --forward-mode=txonly -a
--stats-period 1

*Failing case with 8 rx & 8 tx queues:*
./build/app/dpdk-testpmd -l 0-7 -a 38d9:00:02.0 -- --nb-cores=4 --rxq=8
--txq=8 --burst=64 --mbcache=512 --max-pkt-len=128 --forward-mode=txonly -a
--stats-period 1

*DPDK log:*
Configuring Port 0 (socket 0)
EAL: Cannot send more than 8 FDs
tap_mp_req_on_rxtx(): Failed to send start req to secondary 7
*** stack smashing detected ***: <unknown> terminated
Aborted

=====================================



When we checked DPDK 22.11 code, tap_mp_req_on_rxtx() pasted below is newly
added in rte_eth_tap.c file which is causing the issue.



static int

tap_mp_req_on_rxtx(struct rte_eth_dev *dev)

{

                struct rte_mp_msg msg;

                struct ipc_queues *request_param = (struct ipc_queues
*)msg.param;

                int err;

                int fd_iterator = 0;

                struct pmd_process_private *process_private =
dev->process_private;

                int i;



                memset(&msg, 0, sizeof(msg));

                strlcpy(msg.name, TAP_MP_REQ_START_RXTX, sizeof(msg.name));

                strlcpy(request_param->port_name, dev->data->name,
sizeof(request_param->port_name));

                msg.len_param = sizeof(*request_param);

                for (i = 0; i < dev->data->nb_tx_queues; i++) {

                                msg.fds[fd_iterator++] =
process_private->txq_fds[i];

                                msg.num_fds++;

                                request_param->txq_count++;

                }

                for (i = 0; i < dev->data->nb_rx_queues; i++) {

                                msg.fds[fd_iterator++] =
process_private->rxq_fds[i];

                                msg.num_fds++;

                                request_param->rxq_count++;

                }



                err = rte_mp_sendmsg(&msg);  ========================è This
function is sending msg with msg.num_fds > RTE_PMD_TAP_MAX_QUEUES (i.e. 8),
because of that its failing.

                if (err < 0) {

                                TAP_LOG(ERR, "Failed to send start req to
secondary %d",

                                                rte_errno);

                                return -1;

                }



                return 0;

}



Thanks

Nagesh

[-- Attachment #2: Type: text/html, Size: 9110 bytes --]

Re: DPDK22 issue: Unable to set more than 4 queues in Azure

[Stephen Hemminger]

On Thu, 22 Jun 2023 22:06:10 +0530
Nageswara Rao <nagpen75@gmail.com> wrote:

> Hi All,
> 
> We are observing the following issue with DPDK22.11. We didn’t find any
> upstream patches for this issue on the DPDK github. Is there any known
> issue, please let us know.
> 
> 
> 
> *Issue:*
> 
> On Azure platform, we are unable to configure more than 4 queues. When we
> try to configure more than 4 queues its failing with “EAL: Cannot send more
> than 8 FDs” error.
> 
> Here I am pasting the working and failing testpmd logs.
> 
> Please note that this issue is not observed in DPDK 21.11.
> 

You should be using the native netvsc PMD, not the vdev_netvsc,failsafe,tap kludge.

I don't work on Azure any more but I suspect the issue is that the default
in the kernel for TAP is for the number of queues == number of cores.

You aren't going to see any real benefit from having more queues than
the number of DPDK cores.

Re: DPDK22 issue: Unable to set more than 4 queues in Azure

[Nageswara Rao]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

Yes, we are using vdev_netvsc. Native netvsc PMD we are observing issues in
enabling multiple queues.
Though we have 6 DPDK cores, unable to configure more than 4 queues.

On Mon, Jun 26, 2023 at 12:18 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Thu, 22 Jun 2023 22:06:10 +0530
> Nageswara Rao <nagpen75@gmail.com> wrote:
>
> > Hi All,
> >
> > We are observing the following issue with DPDK22.11. We didn’t find any
> > upstream patches for this issue on the DPDK github. Is there any known
> > issue, please let us know.
> >
> >
> >
> > *Issue:*
> >
> > On Azure platform, we are unable to configure more than 4 queues. When we
> > try to configure more than 4 queues its failing with “EAL: Cannot send
> more
> > than 8 FDs” error.
> >
> > Here I am pasting the working and failing testpmd logs.
> >
> > Please note that this issue is not observed in DPDK 21.11.
> >
>
> You should be using the native netvsc PMD, not the
> vdev_netvsc,failsafe,tap kludge.
>
> I don't work on Azure any more but I suspect the issue is that the default
> in the kernel for TAP is for the number of queues == number of cores.
>
> You aren't going to see any real benefit from having more queues than
> the number of DPDK cores.
>
>

[-- Attachment #2: Type: text/html, Size: 1778 bytes --]

Re: DPDK22 issue: Unable to set more than 4 queues in Azure

[Stephen Hemminger]

On Mon, 26 Jun 2023 15:53:12 +0530
Nageswara Rao <nagpen75@gmail.com> wrote:

> Yes, we are using vdev_netvsc. Native netvsc PMD we are observing issues in
> enabling multiple queues.
> Though we have 6 DPDK cores, unable to configure more than 4 queues.

Check with Long who now maintains the netvsc driver.

For tap, check the parameters of the kernel side tap device (with ethtool).
Also, it would help to see full debug log at startup.

Re: DPDK22 issue: Unable to set more than 4 queues in Azure

[Nageswara Rao]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

Issue is because of tap_mp_req_on_rxtx() sending msg.num_fds more
than RTE_MP_MAX_FD_NUM fds for queues > 4.
Looks like these tap driver changes are added as part of the below patch
http://patches.dpdk.org/project/dpdk/patch/20220121042944.23929-1-kumaraparamesh92@gmail.com/


On Mon, Jun 26, 2023 at 3:53 PM Nageswara Rao <nagpen75@gmail.com> wrote:

> Yes, we are using vdev_netvsc. Native netvsc PMD we are observing issues
> in enabling multiple queues.
> Though we have 6 DPDK cores, unable to configure more than 4 queues.
>
> On Mon, Jun 26, 2023 at 12:18 AM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
>
>> On Thu, 22 Jun 2023 22:06:10 +0530
>> Nageswara Rao <nagpen75@gmail.com> wrote:
>>
>> > Hi All,
>> >
>> > We are observing the following issue with DPDK22.11. We didn’t find any
>> > upstream patches for this issue on the DPDK github. Is there any known
>> > issue, please let us know.
>> >
>> >
>> >
>> > *Issue:*
>> >
>> > On Azure platform, we are unable to configure more than 4 queues. When
>> we
>> > try to configure more than 4 queues its failing with “EAL: Cannot send
>> more
>> > than 8 FDs” error.
>> >
>> > Here I am pasting the working and failing testpmd logs.
>> >
>> > Please note that this issue is not observed in DPDK 21.11.
>> >
>>
>> You should be using the native netvsc PMD, not the
>> vdev_netvsc,failsafe,tap kludge.
>>
>> I don't work on Azure any more but I suspect the issue is that the default
>> in the kernel for TAP is for the number of queues == number of cores.
>>
>> You aren't going to see any real benefit from having more queues than
>> the number of DPDK cores.
>>
>>

[-- Attachment #2: Type: text/html, Size: 2593 bytes --]