[dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni]

Hi,

I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?

Thanks
Anjali

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni]

Pls keep me on cc for response. Thanks!!

Hi,

I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?

Thanks
Anjali

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni]

Hi,

Can someone please help with this? How can packet flow be traced on DPDK? Where are all the logs stored if we do enable the _DEBUG configs in config/ directory?

Anjali

From: Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Date: Tuesday, June 16, 2015 at 9:10 AM
To: "dev@dpdk.org<mailto:dev@dpdk.org>" <dev@dpdk.org<mailto:dev@dpdk.org>>, Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Cc: Microsoft Office User <sumerj@juniper.net<mailto:sumerj@juniper.net>>
Subject: Re: Debugging SR-IOV related packet flow problem with DPDK

Pls keep me on cc for response. Thanks!!

Hi,

I am observing that the SR-IOV enabled NIC does not increment any VF packet counters despite incoming packet flow. I have enabled some debug logs like CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
Can someone point me to where these logs are being printed so I can try to figure out where packets are being dropped? Or how can I debug DPDK packet flow related issues (are there any stats I can look at)?

Thanks
Anjali

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Thomas Monjalon]

Hi,

That's third email in 8 hours for this question.
Do you plan to send an email every 4 hours?

It's really hard to help you without knowing which application you run?
in which VM? which VLAN configuration? etc

2015-06-16 20:22, Anjali Kulkarni:
> Hi,
> 
> Can someone please help with this? How can packet flow be traced on DPDK?
> Where are all the logs stored if we do enable the _DEBUG configs in
> config/ directory?

stdout or somewhere else if you configured logs in your app.

> Anjali
[...]
> I am observing that the SR-IOV enabled NIC does not increment any VF packet
> counters despite incoming packet flow. I have enabled some debug logs like
> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
> Can someone point me to where these logs are being printed so I can try to
> figure out where packets are being dropped? Or how can I debug DPDK packet
> flow related issues (are there any stats I can look at)?

Statistics can be requested through a dedicated ethdev API.

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni]

Thanks for your response.
I am running my own application, and it is actually running inside a
docker, not a VM.
No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
(/dev/uio0 and /dev/uio1) are assigned to the docker.
This worked a while ago perfectly.
What I see is that an ixia connected to the 2 10G NICs is pumping traffic,
but the VF statistics are 0. It is as if the VF or NIC did not see the
packets. 
The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
prints on stdout after enabling the _DEBUG logs in config.

Anjali

On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:

>Hi,
>
>That's third email in 8 hours for this question.
>Do you plan to send an email every 4 hours?
>
>It's really hard to help you without knowing which application you run?
>in which VM? which VLAN configuration? etc
>
>2015-06-16 20:22, Anjali Kulkarni:
>> Hi,
>> 
>> Can someone please help with this? How can packet flow be traced on
>>DPDK?
>> Where are all the logs stored if we do enable the _DEBUG configs in
>> config/ directory?
>
>stdout or somewhere else if you configured logs in your app.
>
>> Anjali
>[...]
>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>packet
>> counters despite incoming packet flow. I have enabled some debug logs
>>like
>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>> Can someone point me to where these logs are being printed so I can try
>>to
>> figure out where packets are being dropped? Or how can I debug DPDK
>>packet
>> flow related issues (are there any stats I can look at)?
>
>Statistics can be requested through a dedicated ethdev API.
>

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Anjali Kulkarni]

Folks,

I am seeing this in the dmesg (VF 0 Hung DMA)

[586307.789098] VF 0 setting Multicast Promiscuos Enable TRUE
[586307.789099] Register before write : 0x9000000
[586307.789100] Register after write : 0x19000000
[586307.802585] dmar: DRHD: handling fault status reg 502
[586307.802597] dmar: DMAR:[DMA Read] Request device [06:10.0] fault addr
7c820000


[563607.533814] DMAR:[fault reason 02] Present bit in context entry is
clear
[563607.544246] ixgbe 0000:06:00.1 int3: VF 0 Hung DMA
[563608.344639] docker0: port 1(veth261ed02) entered forwarding state
[563711.695571] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695578] Register before write : 0x19000000
[563711.695581] Register after write : 0x9000000
[563711.695752] VF 0 setting Multicast Promiscuos Enable FALSE
[563711.695758] Register before write : 0x19000000
[563711.695761] Register after write : 0x9000000

I also see some bootp and ipv6 traffic when I do tcpdump on the PF NIC.

I am thinking perhaps DPDK has not completely taken over the nIC, and
perhaps the linux driver is also somehow interacting?

For intents, my DPDK app is running on the host (actually inside docker)
and NOT inside a VM.

Anjali

On 6/16/15, 2:08 PM, "Anjali Kulkarni" <anjali@juniper.net> wrote:

>Thanks for your response.
>I am running my own application, and it is actually running inside a
>docker, not a VM.
>No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
>(/dev/uio0 and /dev/uio1) are assigned to the docker.
>This worked a while ago perfectly.
>What I see is that an ixia connected to the 2 10G NICs is pumping traffic,
>but the VF statistics are 0. It is as if the VF or NIC did not see the
>packets. 
>The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
>prints on stdout after enabling the _DEBUG logs in config.
>
>Anjali
>
>On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>
>>Hi,
>>
>>That's third email in 8 hours for this question.
>>Do you plan to send an email every 4 hours?
>>
>>It's really hard to help you without knowing which application you run?
>>in which VM? which VLAN configuration? etc
>>
>>2015-06-16 20:22, Anjali Kulkarni:
>>> Hi,
>>> 
>>> Can someone please help with this? How can packet flow be traced on
>>>DPDK?
>>> Where are all the logs stored if we do enable the _DEBUG configs in
>>> config/ directory?
>>
>>stdout or somewhere else if you configured logs in your app.
>>
>>> Anjali
>>[...]
>>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>>packet
>>> counters despite incoming packet flow. I have enabled some debug logs
>>>like
>>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>>> Can someone point me to where these logs are being printed so I can try
>>>to
>>> figure out where packets are being dropped? Or how can I debug DPDK
>>>packet
>>> flow related issues (are there any stats I can look at)?
>>
>>Statistics can be requested through a dedicated ethdev API.
>>
>

Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with DPDK

[Andrew Harvey (agh)]

Please type the following:

dmesg | grep DMAR.  I think you will find that rebinding the uio driver
has caused issues, uio know nothing about the iommu.  If you can you may
have better luck with the vfio driver.

HTH ― Andy

On 6/17/15, 3:52 AM, "dev on behalf of Anjali Kulkarni"
<dev-bounces@dpdk.org on behalf of anjali@juniper.net> wrote:

>Folks,
>
>I am seeing this in the dmesg (VF 0 Hung DMA)
>
>[586307.789098] VF 0 setting Multicast Promiscuos Enable TRUE
>[586307.789099] Register before write : 0x9000000
>[586307.789100] Register after write : 0x19000000
>[586307.802585] dmar: DRHD: handling fault status reg 502
>[586307.802597] dmar: DMAR:[DMA Read] Request device [06:10.0] fault addr
>7c820000
>
>
>[563607.533814] DMAR:[fault reason 02] Present bit in context entry is
>clear
>[563607.544246] ixgbe 0000:06:00.1 int3: VF 0 Hung DMA
>[563608.344639] docker0: port 1(veth261ed02) entered forwarding state
>[563711.695571] VF 0 setting Multicast Promiscuos Enable FALSE
>[563711.695578] Register before write : 0x19000000
>[563711.695581] Register after write : 0x9000000
>[563711.695752] VF 0 setting Multicast Promiscuos Enable FALSE
>[563711.695758] Register before write : 0x19000000
>[563711.695761] Register after write : 0x9000000
>
>I also see some bootp and ipv6 traffic when I do tcpdump on the PF NIC.
>
>I am thinking perhaps DPDK has not completely taken over the nIC, and
>perhaps the linux driver is also somehow interacting?
>
>For intents, my DPDK app is running on the host (actually inside docker)
>and NOT inside a VM.
>
>Anjali
>
>On 6/16/15, 2:08 PM, "Anjali Kulkarni" <anjali@juniper.net> wrote:
>
>>Thanks for your response.
>>I am running my own application, and it is actually running inside a
>>docker, not a VM.
>>No VLAN configuration. Just SR-IOV is enabled on 2 NICs, and both devices
>>(/dev/uio0 and /dev/uio1) are assigned to the docker.
>>This worked a while ago perfectly.
>>What I see is that an ixia connected to the 2 10G NICs is pumping
>>traffic,
>>but the VF statistics are 0. It is as if the VF or NIC did not see the
>>packets. 
>>The source dst MAC is set correctly to the VF¹s MAC. I don¹t see any
>>prints on stdout after enabling the _DEBUG logs in config.
>>
>>Anjali
>>
>>On 6/16/15, 1:52 PM, "Thomas Monjalon" <thomas.monjalon@6wind.com> wrote:
>>
>>>Hi,
>>>
>>>That's third email in 8 hours for this question.
>>>Do you plan to send an email every 4 hours?
>>>
>>>It's really hard to help you without knowing which application you run?
>>>in which VM? which VLAN configuration? etc
>>>
>>>2015-06-16 20:22, Anjali Kulkarni:
>>>> Hi,
>>>> 
>>>> Can someone please help with this? How can packet flow be traced on
>>>>DPDK?
>>>> Where are all the logs stored if we do enable the _DEBUG configs in
>>>> config/ directory?
>>>
>>>stdout or somewhere else if you configured logs in your app.
>>>
>>>> Anjali
>>>[...]
>>>> I am observing that the SR-IOV enabled NIC does not increment any VF
>>>>packet
>>>> counters despite incoming packet flow. I have enabled some debug logs
>>>>like
>>>> CONFIG_RTE_LIBRTE_IXGBE_DEBUG_RX in config file.
>>>> Can someone point me to where these logs are being printed so I can
>>>>try
>>>>to
>>>> figure out where packets are being dropped? Or how can I debug DPDK
>>>>packet
>>>> flow related issues (are there any stats I can look at)?
>>>
>>>Statistics can be requested through a dedicated ethdev API.
>>>
>>
>