[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

All,

I have a Xeon machine with linux kernel 3.14 and dpdk 2.0, I have 6 ports
of which 4 are igb and 2 are ixgbe drivers .. I use the python script in
the dpdk distribution to bind and unbind the ports. I unbind them from
linux and bind to igb_uio, dpdk works just fine, then after a while I
unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of
the 4 igb ports (random) stops responding to ethtool, ethtool bails out
with some error. But otherwise the interface seems to work fine, it has a
linux interface created and pops up in /sys/class/net etc.. Has anyone seen
this before ? I thought of checking before starting to debug this further

Rgds,
Gopa.

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Remy Horton]

Morning,

On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote:
[..]
> then after a while I
> unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of
> the 4 igb ports (random) stops responding to ethtool, ethtool bails out
> with some error. But otherwise the interface seems to work fine, it has a
> linux interface created and pops up in /sys/class/net etc.. Has anyone seen
> this before ? I thought of checking before starting to debug this further

Can you give details of the error? If you were you unbinding from 
igb_uio while examples/ethtool was still running it likley caused 
something to trip up, as at least DPDK ethtool itself was not made with 
run-time unbinding in mind.

Regards,

..Remy

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

Hi Remy,

Thanks for the response. The error is "No such device", some snippets
below. And no I was not using the dpdk ethtool

ge8----->   06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
Network Connection (rev 01)

root:~# ls /sys/class/net/ge8/device/driver/module/drivers/
pci:igb
root:~#

root:~# ethtool ge8
Settings for ge8:
Cannot get device settings: No such device
Cannot get wake-on-lan settings: No such device
Cannot get message level: No such device
Cannot get link status: No such device
No data available

Rgds,
Gopa.

On Thu, Apr 14, 2016 at 12:15 AM, Remy Horton <remy.horton@intel.com> wrote:

> Morning,
>
> On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote:
> [..]
>
>> then after a while I
>> unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of
>> the 4 igb ports (random) stops responding to ethtool, ethtool bails out
>> with some error. But otherwise the interface seems to work fine, it has a
>> linux interface created and pops up in /sys/class/net etc.. Has anyone
>> seen
>> this before ? I thought of checking before starting to debug this further
>>
>
> Can you give details of the error? If you were you unbinding from igb_uio
> while examples/ethtool was still running it likley caused something to trip
> up, as at least DPDK ethtool itself was not made with run-time unbinding in
> mind.
>
> Regards,
>
> ..Remy
>

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Remy Horton]

On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote:
[..]
> ge8----->   06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
> Network Connection (rev 01)
>
> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/
> pci:igb
> root:~#
>
> root:~# ethtool ge8
> Settings for ge8:
> Cannot get device settings: No such device
> Cannot get wake-on-lan settings: No such device
> Cannot get message level: No such device
> Cannot get link status: No such device
> No data available

Seems a little odd. Does dmesg show anything related to igb/ixgbe when 
you try this?

..Remy

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

Nothing in dmesg .. The ethtool was just a side-observation, the biggest
problem was that after unbinding from igb_uio and rebinding to igb, if I
follow it up with an /etc/init.d/network restart, that completely hoses the
linux system - anyone trying to open a socket (ifconfig for example) just
hangs. Thats how I started troubleshooting this and happened to see this
ethtool thing along with it, not sure if its related. Also the issue
doesn't happen with one or two interfaces, there needs to be at least five
or six interfaces for this to happen.

The other thing I noticed is that if I put some sleep (2 seconds) between
unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt
happen, but the ethtool issue still remains

Rgds,
Gopa.

On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com> wrote:

> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote:
> [..]
>
>> ge8----->   06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>> Network Connection (rev 01)
>>
>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/
>> pci:igb
>> root:~#
>>
>> root:~# ethtool ge8
>> Settings for ge8:
>> Cannot get device settings: No such device
>> Cannot get wake-on-lan settings: No such device
>> Cannot get message level: No such device
>> Cannot get link status: No such device
>> No data available
>>
>
> Seems a little odd. Does dmesg show anything related to igb/ixgbe when you
> try this?
>
> ..Remy
>

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

So looks like I figured it out .. I came across this bug reference
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of
checking out my problem with ACPI turned off. And with ACPI turned off, the
problem doesnt happen. So theres something that the igb driver is not happy
about when acpi is on .. Any thoughts ?

Rgds,
Gopa.

On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni <
gopakumar.c.e@gmail.com> wrote:

> Nothing in dmesg .. The ethtool was just a side-observation, the biggest
> problem was that after unbinding from igb_uio and rebinding to igb, if I
> follow it up with an /etc/init.d/network restart, that completely hoses the
> linux system - anyone trying to open a socket (ifconfig for example) just
> hangs. Thats how I started troubleshooting this and happened to see this
> ethtool thing along with it, not sure if its related. Also the issue
> doesn't happen with one or two interfaces, there needs to be at least five
> or six interfaces for this to happen.
>
> The other thing I noticed is that if I put some sleep (2 seconds) between
> unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt
> happen, but the ethtool issue still remains
>
> Rgds,
> Gopa.
>
> On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com>
> wrote:
>
>> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote:
>> [..]
>>
>>> ge8----->   06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>>> Network Connection (rev 01)
>>>
>>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/
>>> pci:igb
>>> root:~#
>>>
>>> root:~# ethtool ge8
>>> Settings for ge8:
>>> Cannot get device settings: No such device
>>> Cannot get wake-on-lan settings: No such device
>>> Cannot get message level: No such device
>>> Cannot get link status: No such device
>>> No data available
>>>
>>
>> Seems a little odd. Does dmesg show anything related to igb/ixgbe when
>> you try this?
>>
>> ..Remy
>>
>
>

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

Well, I jumped to a conclusion too soon on the ACPI, that was a wrong
statement (wishful thinking), I recreated the issue even with ACPI off

This time the problem statement is more narrowed down.

1. dpdk is enabled on the interface, interfaces bound to igb_uio
3. kill the process using dpdk
3. rmmod rte_kni
4. rmmod igb_uio
5. bind interface to igb
6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, and
then stops working

At step  #6, if I do an lspci of the device, the device is completely shut
down (attached a sample lspci output - memory not initialized, irqs not
present in /proc/interrupts etc..). But theres nothing in the dmesg that
shows any kind of errors after the messages about the interface being bound
in step #5

And the wierd part is that the device is up at step #6 for like 30 seconds
before it appears shut down in the pci output. Another observation is that
in step #4 when igb_uio relinquishes control of the device it still seems
to leave it in an initialized state, so the only theory I can think of is
whether the device being left in an initia;ized state when it was handed
off to igb in step #5 caused igb to run into some error and shut down the
device ? The next step is to enable debugs in igb driver and see if there
are any debugs that tells more about what happened

Rgds,
Gopa.



On Fri, Apr 15, 2016 at 12:31 PM, Gopakumar Choorakkot Edakkunni <
gopakumar.c.e@gmail.com> wrote:

> So looks like I figured it out .. I came across this bug reference
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of
> checking out my problem with ACPI turned off. And with ACPI turned off, the
> problem doesnt happen. So theres something that the igb driver is not happy
> about when acpi is on .. Any thoughts ?
>
> Rgds,
> Gopa.
>
> On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni <
> gopakumar.c.e@gmail.com> wrote:
>
>> Nothing in dmesg .. The ethtool was just a side-observation, the biggest
>> problem was that after unbinding from igb_uio and rebinding to igb, if I
>> follow it up with an /etc/init.d/network restart, that completely hoses the
>> linux system - anyone trying to open a socket (ifconfig for example) just
>> hangs. Thats how I started troubleshooting this and happened to see this
>> ethtool thing along with it, not sure if its related. Also the issue
>> doesn't happen with one or two interfaces, there needs to be at least five
>> or six interfaces for this to happen.
>>
>> The other thing I noticed is that if I put some sleep (2 seconds) between
>> unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt
>> happen, but the ethtool issue still remains
>>
>> Rgds,
>> Gopa.
>>
>> On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com>
>> wrote:
>>
>>> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote:
>>> [..]
>>>
>>>> ge8----->   06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>>>> Network Connection (rev 01)
>>>>
>>>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/
>>>> pci:igb
>>>> root:~#
>>>>
>>>> root:~# ethtool ge8
>>>> Settings for ge8:
>>>> Cannot get device settings: No such device
>>>> Cannot get wake-on-lan settings: No such device
>>>> Cannot get message level: No such device
>>>> Cannot get link status: No such device
>>>> No data available
>>>>
>>>
>>> Seems a little odd. Does dmesg show anything related to igb/ixgbe when
>>> you try this?
>>>
>>> ..Remy
>>>
>>
>>
>

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Remy Horton]

On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote:
> This time the problem statement is more narrowed down.
>
> 1. dpdk is enabled on the interface, interfaces bound to igb_uio
> 3. kill the process using dpdk
> 3. rmmod rte_kni
> 4. rmmod igb_uio
> 5. bind interface to igb
> 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds,
> and then stops working

Hmm.. can you try that but with rte_kni left out completely? KNI hooks 
into the Linux network stack and think it at the least needs eliminating 
as a casual factor. Can you also try using uio_pci_generic rather than 
igb_uio?

Those aside, I'm suspecting driver issues, so seeing if I can get one of 
the driver test guys to have a look at this..


Regards,

..Remy

Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk

[Gopakumar Choorakkot Edakkunni]

Just to update this thread. With power management for IGB disabled, this
problem is not seen .. So for now thats the "workaround"

Rgds,
Gopa.

On Mon, Apr 18, 2016 at 3:08 AM, Remy Horton <remy.horton@intel.com> wrote:

>
> On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote:
>
>> This time the problem statement is more narrowed down.
>>
>> 1. dpdk is enabled on the interface, interfaces bound to igb_uio
>> 3. kill the process using dpdk
>> 3. rmmod rte_kni
>> 4. rmmod igb_uio
>> 5. bind interface to igb
>> 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds,
>> and then stops working
>>
>
> Hmm.. can you try that but with rte_kni left out completely? KNI hooks
> into the Linux network stack and think it at the least needs eliminating as
> a casual factor. Can you also try using uio_pci_generic rather than igb_uio?
>
> Those aside, I'm suspecting driver issues, so seeing if I can get one of
> the driver test guys to have a look at this..
>
>
> Regards,
>
> ..Remy
>