Multiple Tx-Queues not working as expected

Multiple Tx-Queues not working as expected

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

Hi Team,

I am trying to set-up 8 Tx-Queues (but only 1 Rx-Queue) and burst traffic
out of different Tx-Queues from the same cpu core on an ixgbe nic (10G).
Although transmitted packets reach the peer, reading the statistics
indicates only the Tx-q[0] has non-zero packets and bytes count, the rest
of the Tx-q[1] to Tx-q[7] all have zero packets and bytes count.

I am following the below sequence.

1. configuration
rte_eth_dev_configure(port-id, 1 /* only 1 rx-q */, 8 /* tx-queues */,
&dev_conf /* memset of 0 on this dev_conf done */);

2. tx queue set-up
struct rte_eth_dev_info dev_info;
rte_eth_dev_info_get(port-id, &dev_info);

struct rte_eth_txconf tx_conf;
memcpy(&tx_conf, &dev_info.default_txconf, sizeof tx_conf);
for(i=0; i<8; i++) {
rte_eth_tx_queue_setup(port-id, i /* queue-id */, 1024 /* num_of_txdesc */,
numa_node, &tx_conf;)
}

3. rx queue set-up
rte_eth_rx_queue_setup(port-id, 0, 1024 /* num_of_rxdesc */, &rx_conf,
mbuf_pool);

4. start the port
rte_eth_dev_start(port-id);

5. call the below tx burst function with different queue-id (range 0 to 7)
every-time for every burst of new packets to be transmitted
rte_eth_tx_burst(port-id, queue-id, pkts_burst, nb_pkts_burst);

6. read stats using rte_eth_xstats_get and verified that only the first
tx-q has non-zero packets and bytes count, rest of the tx-q's have 0
packets and bytes count.

What could be wrong here? Appreciate any help.

Thanks & Regards,
Rajasekhar
ReplyForward

[-- Attachment #2: Type: text/html, Size: 5722 bytes --]

Re: Multiple Tx-Queues not working as expected

[Stephen Hemminger]

On Thu, 16 Feb 2023 23:40:27 +0530
Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:

> Hi Team,
> 
> I am trying to set-up 8 Tx-Queues (but only 1 Rx-Queue) and burst traffic
> out of different Tx-Queues from the same cpu core on an ixgbe nic (10G).
> Although transmitted packets reach the peer, reading the statistics
> indicates only the Tx-q[0] has non-zero packets and bytes count, the rest
> of the Tx-q[1] to Tx-q[7] all have zero packets and bytes count.
> 
> I am following the below sequence.
> 
> 1. configuration
> rte_eth_dev_configure(port-id, 1 /* only 1 rx-q */, 8 /* tx-queues */,
> &dev_conf /* memset of 0 on this dev_conf done */);
> 
> 2. tx queue set-up
> struct rte_eth_dev_info dev_info;
> rte_eth_dev_info_get(port-id, &dev_info);
> 
> struct rte_eth_txconf tx_conf;
> memcpy(&tx_conf, &dev_info.default_txconf, sizeof tx_conf);
> for(i=0; i<8; i++) {
> rte_eth_tx_queue_setup(port-id, i /* queue-id */, 1024 /* num_of_txdesc */,
> numa_node, &tx_conf;)
> }
> 
> 3. rx queue set-up
> rte_eth_rx_queue_setup(port-id, 0, 1024 /* num_of_rxdesc */, &rx_conf,
> mbuf_pool);
> 
> 4. start the port
> rte_eth_dev_start(port-id);
> 
> 5. call the below tx burst function with different queue-id (range 0 to 7)
> every-time for every burst of new packets to be transmitted
> rte_eth_tx_burst(port-id, queue-id, pkts_burst, nb_pkts_burst);
> 
> 6. read stats using rte_eth_xstats_get and verified that only the first
> tx-q has non-zero packets and bytes count, rest of the tx-q's have 0
> packets and bytes count.
> 
> What could be wrong here? Appreciate any help.
> 
> Thanks & Regards,
> Rajasekhar
> ReplyForward

Depending on the type of NIC, some share a signal completion channel
for both RX and TX. If the Rx channel is not polled, than transmit completions
may not happen.

Re: Multiple Tx-Queues not working as expected

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 2370 bytes --]

Ok Stephen, thanks for the information, I can try that.

One of the problems I see with single Tx Queue mode is that Ixia reports
packet drops, though I confirmed with the help of counters (before invoking
tx burst) that all packets are being sent-out. Dumping HW counters don't
report any drops in TX.
Is there a mechanism in DPDK to debug this?

Thanks & Regards,
Rajasekhar

On Fri, Feb 17, 2023 at 1:21 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Thu, 16 Feb 2023 23:40:27 +0530
> Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:
>
> > Hi Team,
> >
> > I am trying to set-up 8 Tx-Queues (but only 1 Rx-Queue) and burst traffic
> > out of different Tx-Queues from the same cpu core on an ixgbe nic (10G).
> > Although transmitted packets reach the peer, reading the statistics
> > indicates only the Tx-q[0] has non-zero packets and bytes count, the rest
> > of the Tx-q[1] to Tx-q[7] all have zero packets and bytes count.
> >
> > I am following the below sequence.
> >
> > 1. configuration
> > rte_eth_dev_configure(port-id, 1 /* only 1 rx-q */, 8 /* tx-queues */,
> > &dev_conf /* memset of 0 on this dev_conf done */);
> >
> > 2. tx queue set-up
> > struct rte_eth_dev_info dev_info;
> > rte_eth_dev_info_get(port-id, &dev_info);
> >
> > struct rte_eth_txconf tx_conf;
> > memcpy(&tx_conf, &dev_info.default_txconf, sizeof tx_conf);
> > for(i=0; i<8; i++) {
> > rte_eth_tx_queue_setup(port-id, i /* queue-id */, 1024 /* num_of_txdesc
> */,
> > numa_node, &tx_conf;)
> > }
> >
> > 3. rx queue set-up
> > rte_eth_rx_queue_setup(port-id, 0, 1024 /* num_of_rxdesc */, &rx_conf,
> > mbuf_pool);
> >
> > 4. start the port
> > rte_eth_dev_start(port-id);
> >
> > 5. call the below tx burst function with different queue-id (range 0 to
> 7)
> > every-time for every burst of new packets to be transmitted
> > rte_eth_tx_burst(port-id, queue-id, pkts_burst, nb_pkts_burst);
> >
> > 6. read stats using rte_eth_xstats_get and verified that only the first
> > tx-q has non-zero packets and bytes count, rest of the tx-q's have 0
> > packets and bytes count.
> >
> > What could be wrong here? Appreciate any help.
> >
> > Thanks & Regards,
> > Rajasekhar
> > ReplyForward
>
> Depending on the type of NIC, some share a signal completion channel
> for both RX and TX. If the Rx channel is not polled, than transmit
> completions
> may not happen.
>

[-- Attachment #2: Type: text/html, Size: 3135 bytes --]

Re: Multiple Tx-Queues not working as expected

[Stephen Hemminger]

On Fri, 17 Feb 2023 11:30:14 +0530
Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:

> Ok Stephen, thanks for the information, I can try that.
> 
> One of the problems I see with single Tx Queue mode is that Ixia reports
> packet drops, though I confirmed with the help of counters (before invoking
> tx burst) that all packets are being sent-out. Dumping HW counters don't
> report any drops in TX.
> Is there a mechanism in DPDK to debug this?
> 
> Thanks & Regards,
> Rajasekhar

The common usage in DPDK is to have one transmit queue per DPDK thread doing
transmits.  If the transmit queue is not configured with enough depth.

The most common packet flow in DPDK is packets getting received, modified
then transmitted. With that pattern, packet loss is at the receiver when
the CPU can't keep up with the packet rate arriving.

Re: Multiple Tx-Queues not working as expected

[Bruce Richardson]

On Fri, Feb 17, 2023 at 11:30:14AM +0530, Rajasekhar Pulluru wrote:
>    Ok Stephen, thanks for the information, I can try that.
>    One of the problems I see with single Tx Queue mode is that Ixia
>    reports packet drops, though I confirmed with the help of counters
>    (before invoking tx burst) that all packets are being sent-out. Dumping
>    HW counters don't report any drops in TX.
>    Is there a mechanism in DPDK to debug this?
>    Thanks & Regards,
>    Rajasekhar
> 
Hi,

so long as the packets are written successfully to the TX ring, they should
be send out ok - unless the actual packets are some way invalid, e.g.
undersized. Are the tx_burst calls reporting that all packets are getting
written to the ring?  All packets successfully written should be reported
as received at the other end.

In terms of the NIC TX stats, I'm not sure about for the ixgbe driver, but
I think in some cases to get per-queue stats, you needed to set up a
mapping of what queues you wanted to track stats for, as the NIC could only
track a certain number of queues - fewer than that available in HW.  See
function [1]. For tracking transmits per queue, it's generally easier just
to have the app track the successful enqueues to the ring. This is what
testpmd does internally for queue stats, I believe (though for port stats
it reads hardware).

/Bruce

[1] https://doc.dpdk.org/api/rte__ethdev_8h.html#a56fae7e398b289f795a1b6256149c4f3

Re: Multiple Tx-Queues not working as expected

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 1526 bytes --]

Hi Stephen,
Have tx queue len was 1024, increasing it to 2048 hasn't changed the
behaviour. IXIA is the receiver in this case, which is definitely(and
tested) capable of receiving more rate than this.

IXIA-TX --->RX(ixgbe-port0)HOST(ixgbe-port1)TX ---> IXIA-RX
Captured the rx packets count at ixgbe-port0 and tx packets count at
ixgbe-port1. These 2 counters are equal and are also equal to the frames
transmitted by IXIA.
But the IXIA-RX packets count reports a few hundreds-to-few thousands of
packets lost on every run.

Thanks & Regards,
Rajasekhar



On Fri, Feb 17, 2023 at 9:56 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Fri, 17 Feb 2023 11:30:14 +0530
> Rajasekhar Pulluru <pullururajasekhar@gmail.com> wrote:
>
> > Ok Stephen, thanks for the information, I can try that.
> >
> > One of the problems I see with single Tx Queue mode is that Ixia reports
> > packet drops, though I confirmed with the help of counters (before
> invoking
> > tx burst) that all packets are being sent-out. Dumping HW counters don't
> > report any drops in TX.
> > Is there a mechanism in DPDK to debug this?
> >
> > Thanks & Regards,
> > Rajasekhar
>
> The common usage in DPDK is to have one transmit queue per DPDK thread
> doing
> transmits.  If the transmit queue is not configured with enough depth.
>
> The most common packet flow in DPDK is packets getting received, modified
> then transmitted. With that pattern, packet loss is at the receiver when
> the CPU can't keep up with the packet rate arriving.
>

[-- Attachment #2: Type: text/html, Size: 2116 bytes --]

Re: Multiple Tx-Queues not working as expected

[Rajasekhar Pulluru]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

Yes Bruce, rte_eth_tx_burst api returns the same count as nb_pkts (4th-arg)

Thanks for pointing out the information on tracking per-queue stats
mapping, can try this out.

Thanks & Regards,
Rajasekhar

On Fri, Feb 17, 2023 at 10:09 PM Bruce Richardson <
bruce.richardson@intel.com> wrote:

> On Fri, Feb 17, 2023 at 11:30:14AM +0530, Rajasekhar Pulluru wrote:
> >    Ok Stephen, thanks for the information, I can try that.
> >    One of the problems I see with single Tx Queue mode is that Ixia
> >    reports packet drops, though I confirmed with the help of counters
> >    (before invoking tx burst) that all packets are being sent-out.
> Dumping
> >    HW counters don't report any drops in TX.
> >    Is there a mechanism in DPDK to debug this?
> >    Thanks & Regards,
> >    Rajasekhar
> >
> Hi,
>
> so long as the packets are written successfully to the TX ring, they should
> be send out ok - unless the actual packets are some way invalid, e.g.
> undersized. Are the tx_burst calls reporting that all packets are getting
> written to the ring?  All packets successfully written should be reported
> as received at the other end.
>
> In terms of the NIC TX stats, I'm not sure about for the ixgbe driver, but
> I think in some cases to get per-queue stats, you needed to set up a
> mapping of what queues you wanted to track stats for, as the NIC could only
> track a certain number of queues - fewer than that available in HW.  See
> function [1]. For tracking transmits per queue, it's generally easier just
> to have the app track the successful enqueues to the ring. This is what
> testpmd does internally for queue stats, I believe (though for port stats
> it reads hardware).
>
> /Bruce
>
> [1]
> https://doc.dpdk.org/api/rte__ethdev_8h.html#a56fae7e398b289f795a1b6256149c4f3
>

[-- Attachment #2: Type: text/html, Size: 2419 bytes --]