[dpdk-dev] soft lockup calling receive burst

[dpdk-dev] soft lockup calling receive burst

[Joseph Vossen]

hello,

I am working on a dpdk-based app using version 1.6 under RH7.3.  Under varying traffic loads, I will intermittently notice a kernel soft lockup and the RIP provided by the kernel always points to the same MOV instruction in rte_ethdev.h (line #1982).  The stack trace looks like:


		dev = &rte_eth_devices[port_id];
		return (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id], rx_pkts, nb_pkts);


		473176:	0f b7 15 a7 68 38 00		movzwl 0x3868a7(%rip),%edx	# 7f9a24 <max_rx_burst> rte_eth_rx_burst():
		47317d:	0f b6 31				movzbl (%rcx),%esi
		473180:	48 89 f0				mov %rsi,%rax
		473183:	0f b6 71 01			movzbl 0x1(%rcx),%esi
		473187:	48 c1 e0 06			shl $0x6,%rax
		47318b:	48 8b b8 70 ed 83 00		mov 0x83ed70(%rax),%rdi
	---->	473192:	48 8b 0f				mov (%rdi),%rcx
		473195:	48 8b 3c f1				mov (%rcx,%rsi,8),%rdi
		473199:	4c 89 ee				mov %r13,%rsi
		47319c:	ff 90 60 ed 83 00		callq *0x83ed60(%rax)


has any one else seen something like this?

thanks

Re: [dpdk-dev] soft lockup calling receive burst

[Jay Rolette]

On Fri, Mar 20, 2015 at 12:53 PM, Joseph Vossen <jkvossen@vossen.org> wrote:

> hello,
>
> I am working on a dpdk-based app using version 1.6 under RH7.3.  Under
> varying traffic loads, I will intermittently notice a kernel soft lockup
> and the RIP provided by the kernel always points to the same MOV
> instruction in rte_ethdev.h (line #1982).  The stack trace looks like:
>
>
>                 dev = &rte_eth_devices[port_id];
>                 return
> (*dev->rx_pkt_burst)(dev->data->rx_queues[queue_id], rx_pkts, nb_pkts);
>
>
>                 473176: 0f b7 15 a7 68 38 00            movzwl
> 0x3868a7(%rip),%edx      # 7f9a24 <max_rx_burst> rte_eth_rx_burst():
>                 47317d: 0f b6 31                                movzbl
> (%rcx),%esi
>                 473180: 48 89 f0                                mov
> %rsi,%rax
>                 473183: 0f b6 71 01                     movzbl
> 0x1(%rcx),%esi
>                 473187: 48 c1 e0 06                     shl $0x6,%rax
>                 47318b: 48 8b b8 70 ed 83 00            mov
> 0x83ed70(%rax),%rdi
>         ---->   473192: 48 8b 0f                                mov
> (%rdi),%rcx
>                 473195: 48 8b 3c f1                             mov
> (%rcx,%rsi,8),%rdi
>                 473199: 4c 89 ee                                mov
> %r13,%rsi
>                 47319c: ff 90 60 ed 83 00               callq
> *0x83ed60(%rax)
>
>
> has any one else seen something like this?
>
> thanks


I haven't seen that, but there's a soft-lockup in KNI in DPDK 1.6. Here's
what I posted about that one a few weeks ago:

Found the problem. No patch to submit since it's already fixed in later
versions of DPDK, but thought I'd follow up with the details since I'm sure
we aren't the only ones trying to use bleeding-edge versions of DPDK...

In kni_net_rx_normal(), it was calling netif_receive_skb() instead of
netif_rx(). The source for netif_receive_skb() point out that it should
only be called from soft-irq context, which isn't the case for KNI.

As typical, simple fix once you track it down.

Yao-Po Wang's fix:  commit 41a6ebded53982107c1adfc0652d6cc1375a7db9.

Jay

Re: [dpdk-dev] soft lockup calling receive burst

[Joseph Vossen]

[snip]

> I haven't seen that, but there's a soft-lockup in KNI in DPDK 1.6. Here's what I posted about that one a few weeks ago:
> 
> Found the problem. No patch to submit since it's already fixed in later versions of DPDK, but thought I'd follow up with the details since I'm sure we aren't the only ones trying to use bleeding-edge versions of DPDK...
> 
> In kni_net_rx_normal(), it was calling netif_receive_skb() instead of netif_rx(). The source for netif_receive_skb() point out that it should only be called from soft-irq context, which isn't the case for KNI.
> 
> As typical, simple fix once you track it down.
> 
> Yao-Po Wang's fix:  commit 41a6ebded53982107c1adfc0652d6cc1375a7db9.

I noticed that posting, but KNI is not involved here...thanks!