Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni <anjali@juniper.net>]

Folks,

I am seeing this in the dmesg (VF 0 Hung DMA)

[586307.789098] VF 0 setting Multicast Promiscuos Enable TRUE
[586307.789099] Register before write : 0x9000000
[586307.789100] Register after write : 0x19000000
[586307.802585] dmar: DRHD: handling fault status reg 502
[586307.802597] dmar: DMAR:[DMA Read] Request device [06:10.0] fault addr
7c820000


[563607.533814] DMAR:[fault reason 02] Present bit in context entry is
clear
[563607.544246] ixgbe 0000:06:00.1 int3: VF 0 Hung DMA
[563608.344639] docker0: port 1(veth261ed02) entered forwarding state
[563711.695571] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695578] Register before write : 0x19000000
[563711.695581] Register after write : 0x9000000
[563711.695752] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695758] Register before write : 0x19000000
[563711.695761] Register after write : 0x9000000

I also see some bootp and ipv6 traffic when I do tcpdump on the PF NIC.

I am thinking perhaps DPDK has not completely taken over the nIC, and
perhaps the linux driver is also somehow interacting?

For intents, my DPDK app is running on the host (actually inside docker)
and NOT inside a VM.

Anjali

On 6/16/15, 2:08 PM, "Anjali Kulkarni" <anjali@juniper.net> wrote:

>Thanks for your response.
>I am running my own application, and it is actually running inside a
>docker, not a VM.
>No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
>(/dev/uio0 and /dev/uio1) are assigned to the docker.
>This worked a while ago perfectly.
>What I see is that an ixia connected to the 2 10G NICs is pumping traffic,
>but the VF statistics are 0. It is as if the VF or NIC did not see the
>packets. 
>The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
>prints on stdout after enabling the _DEBUG logs in config.
>
>Anjali
>
>On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>
>>Hi,
>>
>>That's third email in 8 hours for this question.
>>Do you plan to send an email every 4 hours?
>>
>>It's really hard to help you without knowing which application you run?
>>in which VM? which VLAN configuration? etc
>>
>>2015-06-16 20:22, Anjali Kulkarni:
>>> Hi,
>>> 
>>> Can someone please help with this? How can packet flow be traced on
>>>DPDK?
>>> Where are all the logs stored if we do enable the _DEBUG configs in
>>> config/ directory?
>>
>>stdout or somewhere else if you configured logs in your app.
>>
>>> Anjali
>>[...]
>>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>>packet
>>> counters despite incoming packet flow. I have enabled some debug logs
>>>like
>>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>>> Can someone point me to where these logs are being printed so I can try
>>>to
>>> figure out where packets are being dropped? Or how can I debug DPDK
>>>packet
>>> flow related issues (are there any stats I can look at)?
>>
>>Statistics can be requested through a dedicated ethdev API.
>>
>