Re: [dpdk-users] BUG: unable to handle kernel paging request

["Wiles, Keith" <keith.wiles@intel.com>]

> On Feb 28, 2017, at 9:42 AM, Sushil Adhikari <sushil446@gmail.com> wrote:
> 
> Since printf is not working here I'm using printk, do you mean whether skb_put(skb, len) fails or the memcpy fails( line 169)? I had separated the line 169 in to two one for skb_put and another just memcpy, and it doesn't fail on skb_put so its memory copy that what causing the fail. Since the memory location of data_skv and the location in "BUG: unable to handle kernel paging request at 000007529d212900" matches I thought the data_skv address is not correct or something.

If you try printing the a byte or word at the data_kva address does the printf fail?

If not try looping on the address read every 128 bytes and see how far you get. If it is the first address then I am guessing the mbuf_kva address is bad. Then we need to look in the MAINTAINERS file and email the maintainer directly to see if he knows what is happening.

> 
> On Tue, Feb 28, 2017 at 9:34 AM, Wiles, Keith <keith.wiles@intel.com> wrote:
> 
> > On Feb 28, 2017, at 9:30 AM, Sushil Adhikari <sushil446@gmail.com> wrote:
> >
> > its failing at data_kva address because this is where I'm getting the kernel paging request fail
> > BUG: unable to handle kernel paging request at 000007529d212900
> > and this is what my debug shows
> > kva data addresses: data_kva 000007529d212900, kva->buff_add 00007f529d212880, kva->data_off 128, kni->mbuf_va  (null), and kni->mbuf_kva ffff880000000000
> 
> I was thinking of using GDB to dump memory or use printf to see which one is failing.
> 
> >
> > I'm not sure how to verify that these are normal
> >
> > On Mon, Feb 27, 2017 at 4:41 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
> >
> > > On Feb 27, 2017, at 4:22 PM, Sushil Adhikari <sushil446@gmail.com> wrote:
> > >
> > > I narrowed it to location where it was failing, its coming from http://dpdk.org/browse/dpdk-stable/tree/lib/librte_eal/linuxapp/kni/kni_net.c?h=v16.07.2 line 169, I am getting the value of len to be 1358 from len=kva->pkt_len; which seems right for ip packet and the memory allocation from line 157 also seems to be working fine. when I print the sizeof(*skb) or sizeof(struct sk_buff) its giving me 208, I guess I dont know whether it should be the size we allocate from line 157, which is len + 2 = 1360 or its fixed size structure of 208 byte. I would appreciate any insight.
> > > Linux version 4.4.20 (sushila@dev03) (gcc version 4.9.2 (crosstool-NG 1.20.0) ) #1 SMP Fri Feb 24 14:32:28 CST 2017
> >
> > Looks like we need to determine which address is failing the skb_put() or data_kva address. If the address that fails is at the end of the skb_put() then I would think the len is wrong, meaning we are stepping on memory just passed a page for the skb. If the address that fails is in the data_kva then the calculations for that address are wrong in line 154. You may want to printout the kva->data_off, buf_addr, mbuf_va and mbuf_kva to verify these values seem normal. The data_off value should be reasonable (I guess) meaning with a 2K range.
> >
> > Also print out the two values skb_put() and data_kva. You can use gdb to example the memory using the dump memory command. (is it x/nn <address>) nn is the read width, but you could leave off the ‘/nn’ for the default.
> >
> > >
> > > Thank you
> > > Sushil
> > >
> > > On Sat, Feb 25, 2017 at 10:31 AM, Wiles, Keith <keith.wiles@intel.com> wrote:
> > >
> > > > On Feb 24, 2017, at 8:07 AM, Sushil Adhikari <sushil446@gmail.com> wrote:
> > > >
> > > > Resending because of unsupported email content type
> > > >
> > > >
> > > > yes hanging is the better word I guess,  ctrl + c is not working to actually stop the program. I also had display connected to the target manchine and I have attached a picture that shows the messages in that display that is where I saw "BUG:Unable to handle kernel paging request at xxxxxx", which made me think that the program is in bad state.
> > >
> > > Sorry, I do not see why you are getting this message. All I can suggest is to use GDB and see if you can determine why the message is happening.
> > >
> > > >
> > > > info thread in gdb shows one thread running
> > > > Id   Target Id         Frame
> > > > * 1    LWP 843 "dpdkKni" 0x000000000044eaee in rte_kni_tx_burst ()
> > > >
> > > > On Thu, Feb 23, 2017 at 5:41 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
> > > >
> > > > > On Feb 23, 2017, at 2:38 PM, Sushil Adhikari <sushil446@gmail.com> wrote:
> > > > >
> > > > > While trying to run dpdk Kni application I ran in to a problem, with
> > > > > following error message
> > > > > BUG: unable to handle kernel paging request at 000007ffe2b92780
> > > > >
> > > > > To run the application I first unbinded the ports from kernel module and
> > > > > binded them to igb_uio
> > > > >> echo 0000:05:00.1 > /sys/bus/pci/drivers/ixgbe/unbind
> > > > >> echo 0000:05:00.0 > /sys/bus/pci/drivers/ixgbe/unbind
> > > > >> echo 0x8086 0x1528 > /sys/bus/pci/drivers/igb_uio/new_id
> > > > >
> > > > > I ran the application using gdb as
> > > > >
> > > > > [~]$ /root/gdb dpdkKni
> > > > > GNU gdb (crosstool-NG 1.20.0) 7.8
> > > > > Copyright (C) 2014 Free Software Foundation, Inc.
> > > > > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
> > > > >>
> > > > > This is free software: you are free to change and redistribute it.
> > > > > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > > > > and "show warranty" for details.
> > > > > This GDB was configured as "x86_64-unknown-linux-gnu".
> > > > > Type "show configuration" for configuration details.
> > > > > For bug reporting instructions, please see:
> > > > > <http://www.gnu.org/software/gdb/bugs/>.
> > > > > Find the GDB manual and other documentation resources online at:
> > > > > <http://www.gnu.org/software/gdb/documentation/>.
> > > > > For help, type "help".
> > > > > Type "apropos word" to search for commands related to "word"...
> > > > > Reading symbols from dpdkKni...(no debugging symbols found)...done.
> > > > > (gdb) Run dpdkKni -c 0x0f -n 4 -- -P -p 0x3 --config="(0,0,1),(1,2,3)"
> > > > > Starting program: /root/dpdkKni dpdkKni -c 0x0f -n 4 -- -P -p 0x3
> > > > > --config="(0,0,1),(1,2,3)"
> > > > > warning: Could not load shared library symbols for linux-vdso.so.1.
> > > > > Do you need "set solib-search-path" or "set sysroot"?
> > > > > warning: Unable to find libthread_db matching inferior's thread library,
> > > > > thread debugging will not be available.
> > > > > EAL: Detected 4 lcore(s)
> > > > > EAL: Probing VFIO support...
> > > > > EAL: PCI device 0000:05:00.0 on NUMA socket -1
> > > > > EAL:   probe driver: 8086:1528 net_ixgbe
> > > > > EAL: PCI device 0000:05:00.1 on NUMA socket -1
> > > > > EAL:   probe driver: 8086:1528 net_ixgbe
> > > > > Address of pktmbuf_pool 0x7ffff5a7dec0
> > > > > APP: Initialising port 0 ...
> > > > > KNI: pci: 05:00:00       8086:1528
> > > > > kni created for port 0 with kni[i] address 0x7fff75638280 with i 0
> > > > > APP: Initialising port 1 ...
> > > > > KNI: pci: 05:00:01       8086:1528
> > > > > kni created for port 1 with kni[i] address 0x7fff75629e00 with i 0
> > > > > APP: Lcore 1 is writing to port 0
> > > > > APP: Lcore 2 is reading from port 1
> > > > > APP: Lcore 3 is writing to port 1
> > > > > APP: Lcore 0 is reading from port 0
> > > > > ^C
> > > > > Program received signal SIGINT, Interrupt.
> > > >
> > > > The program did not crash or get a segfault, but you hit control-c which stopped the application. When you ran the application you started 4 threads and this is why it would appear in different places when stopped.
> > > >
> > > > If the application is hanging then you can use control-C and then do ‘info threads’ command to see the location of all threads. You can use the ‘thread X’ command to switch between threads. Please check the command usage here I am going from memory.
> > > >
> > > > I am not sure if the application has a -i option to get a command line if so that maybe useful to enable, check the application to see if it used cmdline feature.
> > > >
> > > > It maybe the application just sits running and you have to use other tools or apps to send traffic on the KNI application, sorry I have not really used the KNI example.
> > > >
> > > > > 0x000000000044e916 in rte_kni_tx_burst ()
> > > > > (gdb) backtrace
> > > > > #0  0x000000000044e916 in rte_kni_tx_burst ()
> > > > > #1  0x0000000000619758 in main_loop(void*) ()
> > > > > #2  0x0000000000431183 in rte_eal_mp_remote_launch ()
> > > > > #3  0x000000000040d312 in main ()
> > > > >
> > > > > (this is where the program crashes)
> > > > >
> > > > > I tried to trace the crash with gdb(I am new to gdb)
> > > > >
> > > > > and when I do the backtrace it ends up in different functions each time:
> > > > > this time it gave me rte_kni_tx_burst()
> > > > >
> > > > > I'm running latest dpdk version 17.02 and linux kernel is
> > > > > Linux version 4.4.20 (tcuser@cibuild08) (gcc version 4.9.2 (crosstool-NG
> > > > > 1.20.0) ) #1 SMP
> > > > >
> > > > > I would appreciate any suggestion or insight regarding this issue.
> > > >
> > > > Regards,
> > > > Keith
> > > >
> > > >
> > > > <kni.jpg>
> > >
> > > Regards,
> > > Keith
> > >
> > >
> >
> > Regards,
> > Keith
> >
> >
> 
> Regards,
> Keith
> 
> 

Regards,
Keith