* [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk @ 2016-04-13 14:48 Gopakumar Choorakkot Edakkunni 2016-04-14 7:15 ` Remy Horton 0 siblings, 1 reply; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-13 14:48 UTC (permalink / raw) To: dev All, I have a Xeon machine with linux kernel 3.14 and dpdk 2.0, I have 6 ports of which 4 are igb and 2 are ixgbe drivers .. I use the python script in the dpdk distribution to bind and unbind the ports. I unbind them from linux and bind to igb_uio, dpdk works just fine, then after a while I unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of the 4 igb ports (random) stops responding to ethtool, ethtool bails out with some error. But otherwise the interface seems to work fine, it has a linux interface created and pops up in /sys/class/net etc.. Has anyone seen this before ? I thought of checking before starting to debug this further Rgds, Gopa. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-13 14:48 [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk Gopakumar Choorakkot Edakkunni @ 2016-04-14 7:15 ` Remy Horton 2016-04-14 19:25 ` Gopakumar Choorakkot Edakkunni 0 siblings, 1 reply; 9+ messages in thread From: Remy Horton @ 2016-04-14 7:15 UTC (permalink / raw) To: Gopakumar Choorakkot Edakkunni, dev Morning, On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote: [..] > then after a while I > unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of > the 4 igb ports (random) stops responding to ethtool, ethtool bails out > with some error. But otherwise the interface seems to work fine, it has a > linux interface created and pops up in /sys/class/net etc.. Has anyone seen > this before ? I thought of checking before starting to debug this further Can you give details of the error? If you were you unbinding from igb_uio while examples/ethtool was still running it likley caused something to trip up, as at least DPDK ethtool itself was not made with run-time unbinding in mind. Regards, ..Remy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-14 7:15 ` Remy Horton @ 2016-04-14 19:25 ` Gopakumar Choorakkot Edakkunni 2016-04-15 7:50 ` Remy Horton 0 siblings, 1 reply; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-14 19:25 UTC (permalink / raw) To: Remy Horton; +Cc: dev Hi Remy, Thanks for the response. The error is "No such device", some snippets below. And no I was not using the dpdk ethtool ge8-----> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ pci:igb root:~# root:~# ethtool ge8 Settings for ge8: Cannot get device settings: No such device Cannot get wake-on-lan settings: No such device Cannot get message level: No such device Cannot get link status: No such device No data available Rgds, Gopa. On Thu, Apr 14, 2016 at 12:15 AM, Remy Horton <remy.horton@intel.com> wrote: > Morning, > > On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> then after a while I >> unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of >> the 4 igb ports (random) stops responding to ethtool, ethtool bails out >> with some error. But otherwise the interface seems to work fine, it has a >> linux interface created and pops up in /sys/class/net etc.. Has anyone >> seen >> this before ? I thought of checking before starting to debug this further >> > > Can you give details of the error? If you were you unbinding from igb_uio > while examples/ethtool was still running it likley caused something to trip > up, as at least DPDK ethtool itself was not made with run-time unbinding in > mind. > > Regards, > > ..Remy > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-14 19:25 ` Gopakumar Choorakkot Edakkunni @ 2016-04-15 7:50 ` Remy Horton 2016-04-15 17:38 ` Gopakumar Choorakkot Edakkunni 0 siblings, 1 reply; 9+ messages in thread From: Remy Horton @ 2016-04-15 7:50 UTC (permalink / raw) To: Gopakumar Choorakkot Edakkunni; +Cc: dev On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: [..] > ge8-----> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit > Network Connection (rev 01) > > root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ > pci:igb > root:~# > > root:~# ethtool ge8 > Settings for ge8: > Cannot get device settings: No such device > Cannot get wake-on-lan settings: No such device > Cannot get message level: No such device > Cannot get link status: No such device > No data available Seems a little odd. Does dmesg show anything related to igb/ixgbe when you try this? ..Remy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-15 7:50 ` Remy Horton @ 2016-04-15 17:38 ` Gopakumar Choorakkot Edakkunni 2016-04-15 19:31 ` Gopakumar Choorakkot Edakkunni 0 siblings, 1 reply; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-15 17:38 UTC (permalink / raw) To: Remy Horton; +Cc: dev Nothing in dmesg .. The ethtool was just a side-observation, the biggest problem was that after unbinding from igb_uio and rebinding to igb, if I follow it up with an /etc/init.d/network restart, that completely hoses the linux system - anyone trying to open a socket (ifconfig for example) just hangs. Thats how I started troubleshooting this and happened to see this ethtool thing along with it, not sure if its related. Also the issue doesn't happen with one or two interfaces, there needs to be at least five or six interfaces for this to happen. The other thing I noticed is that if I put some sleep (2 seconds) between unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt happen, but the ethtool issue still remains Rgds, Gopa. On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com> wrote: > On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> ge8-----> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >> Network Connection (rev 01) >> >> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >> pci:igb >> root:~# >> >> root:~# ethtool ge8 >> Settings for ge8: >> Cannot get device settings: No such device >> Cannot get wake-on-lan settings: No such device >> Cannot get message level: No such device >> Cannot get link status: No such device >> No data available >> > > Seems a little odd. Does dmesg show anything related to igb/ixgbe when you > try this? > > ..Remy > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-15 17:38 ` Gopakumar Choorakkot Edakkunni @ 2016-04-15 19:31 ` Gopakumar Choorakkot Edakkunni 2016-04-15 22:56 ` Gopakumar Choorakkot Edakkunni 0 siblings, 1 reply; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-15 19:31 UTC (permalink / raw) To: Remy Horton; +Cc: dev So looks like I figured it out .. I came across this bug reference https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of checking out my problem with ACPI turned off. And with ACPI turned off, the problem doesnt happen. So theres something that the igb driver is not happy about when acpi is on .. Any thoughts ? Rgds, Gopa. On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e@gmail.com> wrote: > Nothing in dmesg .. The ethtool was just a side-observation, the biggest > problem was that after unbinding from igb_uio and rebinding to igb, if I > follow it up with an /etc/init.d/network restart, that completely hoses the > linux system - anyone trying to open a socket (ifconfig for example) just > hangs. Thats how I started troubleshooting this and happened to see this > ethtool thing along with it, not sure if its related. Also the issue > doesn't happen with one or two interfaces, there needs to be at least five > or six interfaces for this to happen. > > The other thing I noticed is that if I put some sleep (2 seconds) between > unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt > happen, but the ethtool issue still remains > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com> > wrote: > >> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >> [..] >> >>> ge8-----> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>> Network Connection (rev 01) >>> >>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>> pci:igb >>> root:~# >>> >>> root:~# ethtool ge8 >>> Settings for ge8: >>> Cannot get device settings: No such device >>> Cannot get wake-on-lan settings: No such device >>> Cannot get message level: No such device >>> Cannot get link status: No such device >>> No data available >>> >> >> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >> you try this? >> >> ..Remy >> > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-15 19:31 ` Gopakumar Choorakkot Edakkunni @ 2016-04-15 22:56 ` Gopakumar Choorakkot Edakkunni 2016-04-18 10:08 ` Remy Horton 0 siblings, 1 reply; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-15 22:56 UTC (permalink / raw) To: Remy Horton; +Cc: dev Well, I jumped to a conclusion too soon on the ACPI, that was a wrong statement (wishful thinking), I recreated the issue even with ACPI off This time the problem statement is more narrowed down. 1. dpdk is enabled on the interface, interfaces bound to igb_uio 3. kill the process using dpdk 3. rmmod rte_kni 4. rmmod igb_uio 5. bind interface to igb 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, and then stops working At step #6, if I do an lspci of the device, the device is completely shut down (attached a sample lspci output - memory not initialized, irqs not present in /proc/interrupts etc..). But theres nothing in the dmesg that shows any kind of errors after the messages about the interface being bound in step #5 And the wierd part is that the device is up at step #6 for like 30 seconds before it appears shut down in the pci output. Another observation is that in step #4 when igb_uio relinquishes control of the device it still seems to leave it in an initialized state, so the only theory I can think of is whether the device being left in an initia;ized state when it was handed off to igb in step #5 caused igb to run into some error and shut down the device ? The next step is to enable debugs in igb driver and see if there are any debugs that tells more about what happened Rgds, Gopa. On Fri, Apr 15, 2016 at 12:31 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e@gmail.com> wrote: > So looks like I figured it out .. I came across this bug reference > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of > checking out my problem with ACPI turned off. And with ACPI turned off, the > problem doesnt happen. So theres something that the igb driver is not happy > about when acpi is on .. Any thoughts ? > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < > gopakumar.c.e@gmail.com> wrote: > >> Nothing in dmesg .. The ethtool was just a side-observation, the biggest >> problem was that after unbinding from igb_uio and rebinding to igb, if I >> follow it up with an /etc/init.d/network restart, that completely hoses the >> linux system - anyone trying to open a socket (ifconfig for example) just >> hangs. Thats how I started troubleshooting this and happened to see this >> ethtool thing along with it, not sure if its related. Also the issue >> doesn't happen with one or two interfaces, there needs to be at least five >> or six interfaces for this to happen. >> >> The other thing I noticed is that if I put some sleep (2 seconds) between >> unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt >> happen, but the ethtool issue still remains >> >> Rgds, >> Gopa. >> >> On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton <remy.horton@intel.com> >> wrote: >> >>> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >>> [..] >>> >>>> ge8-----> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>>> Network Connection (rev 01) >>>> >>>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>>> pci:igb >>>> root:~# >>>> >>>> root:~# ethtool ge8 >>>> Settings for ge8: >>>> Cannot get device settings: No such device >>>> Cannot get wake-on-lan settings: No such device >>>> Cannot get message level: No such device >>>> Cannot get link status: No such device >>>> No data available >>>> >>> >>> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >>> you try this? >>> >>> ..Remy >>> >> >> > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-15 22:56 ` Gopakumar Choorakkot Edakkunni @ 2016-04-18 10:08 ` Remy Horton 2016-04-30 1:23 ` Gopakumar Choorakkot Edakkunni 0 siblings, 1 reply; 9+ messages in thread From: Remy Horton @ 2016-04-18 10:08 UTC (permalink / raw) To: Gopakumar Choorakkot Edakkunni; +Cc: dev On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote: > This time the problem statement is more narrowed down. > > 1. dpdk is enabled on the interface, interfaces bound to igb_uio > 3. kill the process using dpdk > 3. rmmod rte_kni > 4. rmmod igb_uio > 5. bind interface to igb > 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, > and then stops working Hmm.. can you try that but with rte_kni left out completely? KNI hooks into the Linux network stack and think it at the least needs eliminating as a casual factor. Can you also try using uio_pci_generic rather than igb_uio? Those aside, I'm suspecting driver issues, so seeing if I can get one of the driver test guys to have a look at this.. Regards, ..Remy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk 2016-04-18 10:08 ` Remy Horton @ 2016-04-30 1:23 ` Gopakumar Choorakkot Edakkunni 0 siblings, 0 replies; 9+ messages in thread From: Gopakumar Choorakkot Edakkunni @ 2016-04-30 1:23 UTC (permalink / raw) To: Remy Horton; +Cc: dev Just to update this thread. With power management for IGB disabled, this problem is not seen .. So for now thats the "workaround" Rgds, Gopa. On Mon, Apr 18, 2016 at 3:08 AM, Remy Horton <remy.horton@intel.com> wrote: > > On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote: > >> This time the problem statement is more narrowed down. >> >> 1. dpdk is enabled on the interface, interfaces bound to igb_uio >> 3. kill the process using dpdk >> 3. rmmod rte_kni >> 4. rmmod igb_uio >> 5. bind interface to igb >> 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, >> and then stops working >> > > Hmm.. can you try that but with rte_kni left out completely? KNI hooks > into the Linux network stack and think it at the least needs eliminating as > a casual factor. Can you also try using uio_pci_generic rather than igb_uio? > > Those aside, I'm suspecting driver issues, so seeing if I can get one of > the driver test guys to have a look at this.. > > > Regards, > > ..Remy > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-04-30 1:23 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-04-13 14:48 [dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk Gopakumar Choorakkot Edakkunni 2016-04-14 7:15 ` Remy Horton 2016-04-14 19:25 ` Gopakumar Choorakkot Edakkunni 2016-04-15 7:50 ` Remy Horton 2016-04-15 17:38 ` Gopakumar Choorakkot Edakkunni 2016-04-15 19:31 ` Gopakumar Choorakkot Edakkunni 2016-04-15 22:56 ` Gopakumar Choorakkot Edakkunni 2016-04-18 10:08 ` Remy Horton 2016-04-30 1:23 ` Gopakumar Choorakkot Edakkunni
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).