[dpdk-dev] [RFC] l2fwd: trying to expose eth_igb_xmit_pkts unusual behavior

[dpdk-dev] [RFC] l2fwd: trying to expose eth_igb_xmit_pkts unusual behavior

[Ciprian Barbu]

From: Ciprian Barbu <ciprian.barbu@enea.com>

This tries to show an approximate behavior described in an earlier discussion
called "can eth_igb_xmit_pkts called with len 0 affect transmission?"

I'm using Intel i350 dual port 1Gb card and I've tweaked the pool size to the
minimum possible. But unmodified l2fwd still worked withouth a sweat. After
applying this patch transmission stopped after maybe 30 seconds. Lowering the
'magic' number causes l2fwd to stop sending packets even earlier.
Using gdb I could see eth_igb_xmit_pkts always returning because nb_tx was 0
and E1000_TXD_STAT_DD was not set (approx igb_rxtx.c : 476)

Signed-off-by: Ciprian Barbu <ciprian.barbu@enea.com>
---
 examples/l2fwd/main.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
index 17621ee..623b836 100644
--- a/examples/l2fwd/main.c
+++ b/examples/l2fwd/main.c
@@ -72,7 +72,7 @@
 #define RTE_LOGTYPE_L2FWD RTE_LOGTYPE_USER1
 
 #define MBUF_SIZE (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
-#define NB_MBUF   8192
+#define NB_MBUF   1024
 
 #define MAX_PKT_BURST 32
 #define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */
@@ -260,7 +260,7 @@ l2fwd_main_loop(void)
 	struct rte_mbuf *m;
 	unsigned lcore_id;
 	uint64_t prev_tsc, diff_tsc, cur_tsc, timer_tsc;
-	unsigned i, j, portid, nb_rx;
+	unsigned i, j, portid, nb_rx, magic = 0;
 	struct lcore_queue_conf *qconf;
 	const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) / US_PER_S * BURST_TX_DRAIN_US;
 
@@ -285,6 +285,15 @@ l2fwd_main_loop(void)
 	}
 
 	while (1) {
+		if (++magic == 50000) {
+			magic = 0;
+			l2fwd_send_burst(&lcore_queue_conf[lcore_id],
+					 0,
+					 (uint8_t) 0);
+			l2fwd_send_burst(&lcore_queue_conf[lcore_id],
+					 0,
+					 (uint8_t) 1);
+		}
 
 		cur_tsc = rte_rdtsc();
 
-- 
1.9.1

Re: [dpdk-dev] [RFC] l2fwd: trying to expose eth_igb_xmit_pkts unusual behavior

[Zoltan Kiss]

Hi,

On 24/07/15 11:13, Ciprian Barbu wrote:
> From: Ciprian Barbu <ciprian.barbu@enea.com>
>
> This tries to show an approximate behavior described in an earlier discussion
> called "can eth_igb_xmit_pkts called with len 0 affect transmission?"
>
> I'm using Intel i350 dual port 1Gb card and I've tweaked the pool size to the
> minimum possible. But unmodified l2fwd still worked withouth a sweat. After
> applying this patch transmission stopped after maybe 30 seconds. Lowering the
> 'magic' number causes l2fwd to stop sending packets even earlier.
> Using gdb I could see eth_igb_xmit_pkts always returning because nb_tx was 0
> and E1000_TXD_STAT_DD was not set (approx igb_rxtx.c : 476)

I think the proper solution for this will be to introduce an explicit 
API which makes it possible for the application to call the actual PMD's 
callback function, which flushes out the completed buffers from the TX 
ring, if possible. Then in case of igb it could be a noop. See the 
discussion in this thread:

http://dpdk.org/ml/archives/dev/2015-June/018487.html

I'll plan to propose such API, but it won't be in the very near future, 
as 2.1 will come out soon and no new features accepted at the moment.
For the short term we can introduce a check in ODP-DPDK, which avoids 
this rte_eth_tx_burst(..., 0) call if the PMD is igb. How about that?


>
> Signed-off-by: Ciprian Barbu <ciprian.barbu@enea.com>
> ---
>   examples/l2fwd/main.c | 13 +++++++++++--
>   1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
> index 17621ee..623b836 100644
> --- a/examples/l2fwd/main.c
> +++ b/examples/l2fwd/main.c
> @@ -72,7 +72,7 @@
>   #define RTE_LOGTYPE_L2FWD RTE_LOGTYPE_USER1
>
>   #define MBUF_SIZE (2048 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
> -#define NB_MBUF   8192
> +#define NB_MBUF   1024
>
>   #define MAX_PKT_BURST 32
>   #define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */
> @@ -260,7 +260,7 @@ l2fwd_main_loop(void)
>   	struct rte_mbuf *m;
>   	unsigned lcore_id;
>   	uint64_t prev_tsc, diff_tsc, cur_tsc, timer_tsc;
> -	unsigned i, j, portid, nb_rx;
> +	unsigned i, j, portid, nb_rx, magic = 0;
>   	struct lcore_queue_conf *qconf;
>   	const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) / US_PER_S * BURST_TX_DRAIN_US;
>
> @@ -285,6 +285,15 @@ l2fwd_main_loop(void)
>   	}
>
>   	while (1) {
> +		if (++magic == 50000) {
> +			magic = 0;
> +			l2fwd_send_burst(&lcore_queue_conf[lcore_id],
> +					 0,
> +					 (uint8_t) 0);
> +			l2fwd_send_burst(&lcore_queue_conf[lcore_id],
> +					 0,
> +					 (uint8_t) 1);
> +		}
>
>   		cur_tsc = rte_rdtsc();
>
>

Re: [dpdk-dev] [RFC] l2fwd: trying to expose eth_igb_xmit_pkts unusual behavior

[ciprian.barbu]

ping

On 24.07.2015 15:59, Zoltan Kiss wrote:
> Hi,
>
> On 24/07/15 11:13, Ciprian Barbu wrote:
>> From: Ciprian Barbu <ciprian.barbu@enea.com>
>>
>> This tries to show an approximate behavior described in an earlier
>> discussion
>> called "can eth_igb_xmit_pkts called with len 0 affect transmission?"
>>
>> I'm using Intel i350 dual port 1Gb card and I've tweaked the pool size
>> to the
>> minimum possible. But unmodified l2fwd still worked withouth a sweat.
>> After
>> applying this patch transmission stopped after maybe 30 seconds.
>> Lowering the
>> 'magic' number causes l2fwd to stop sending packets even earlier.
>> Using gdb I could see eth_igb_xmit_pkts always returning because nb_tx
>> was 0
>> and E1000_TXD_STAT_DD was not set (approx igb_rxtx.c : 476)
>
> I think the proper solution for this will be to introduce an explicit
> API which makes it possible for the application to call the actual PMD's
> callback function, which flushes out the completed buffers from the TX
> ring, if possible. Then in case of igb it could be a noop. See the
> discussion in this thread:

What I' trying to show here is an unexpected behavior in the tx routine 
of the igb pmd. The documentation doesn't mention that making this call 
with 0 nb_pkt can cause undefined behavior. It would be nice if someone 
could take a look at this.

>
> http://dpdk.org/ml/archives/dev/2015-June/018487.html
>
> I'll plan to propose such API, but it won't be in the very near future,
> as 2.1 will come out soon and no new features accepted at the moment.
> For the short term we can introduce a check in ODP-DPDK, which avoids
> this rte_eth_tx_burst(..., 0) call if the PMD is igb. How about that?
>
>
>>
>> Signed-off-by: Ciprian Barbu <ciprian.barbu@enea.com>
>> ---
>>   examples/l2fwd/main.c | 13 +++++++++++--
>>   1 file changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/examples/l2fwd/main.c b/examples/l2fwd/main.c
>> index 17621ee..623b836 100644
>> --- a/examples/l2fwd/main.c
>> +++ b/examples/l2fwd/main.c
>> @@ -72,7 +72,7 @@
>>   #define RTE_LOGTYPE_L2FWD RTE_LOGTYPE_USER1
>>
>>   #define MBUF_SIZE (2048 + sizeof(struct rte_mbuf) +
>> RTE_PKTMBUF_HEADROOM)
>> -#define NB_MBUF   8192
>> +#define NB_MBUF   1024
>>
>>   #define MAX_PKT_BURST 32
>>   #define BURST_TX_DRAIN_US 100 /* TX drain every ~100us */
>> @@ -260,7 +260,7 @@ l2fwd_main_loop(void)
>>       struct rte_mbuf *m;
>>       unsigned lcore_id;
>>       uint64_t prev_tsc, diff_tsc, cur_tsc, timer_tsc;
>> -    unsigned i, j, portid, nb_rx;
>> +    unsigned i, j, portid, nb_rx, magic = 0;
>>       struct lcore_queue_conf *qconf;
>>       const uint64_t drain_tsc = (rte_get_tsc_hz() + US_PER_S - 1) /
>> US_PER_S * BURST_TX_DRAIN_US;
>>
>> @@ -285,6 +285,15 @@ l2fwd_main_loop(void)
>>       }
>>
>>       while (1) {
>> +        if (++magic == 50000) {
>> +            magic = 0;
>> +            l2fwd_send_burst(&lcore_queue_conf[lcore_id],
>> +                     0,
>> +                     (uint8_t) 0);
>> +            l2fwd_send_burst(&lcore_queue_conf[lcore_id],
>> +                     0,
>> +                     (uint8_t) 1);
>> +        }
>>
>>           cur_tsc = rte_rdtsc();
>>
>>