From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ot0-f194.google.com (mail-ot0-f194.google.com [74.125.82.194]) by dpdk.org (Postfix) with ESMTP id 9F1D52BAA for ; Tue, 28 Feb 2017 16:42:22 +0100 (CET) Received: by mail-ot0-f194.google.com with SMTP id w44so1492234otw.1 for ; Tue, 28 Feb 2017 07:42:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=5C7GrvmBCtU7hlvWvFE8nf5I6UlP1swgVhRZJ44sGqI=; b=d593sibdiQ/iBo6gdpX99S/4+uo10d4h3OdKvcjB0tdM4bKbR4RM/SYXDWpaL8Tx6p xAyLqqmDClx1G5Z983i/e5pq4P+VcR9eH4VtE9dNMeC0C+If6SEz/9g1+LBe7VzQZpUk 0nQ2NuP3WRKKW3MHgdjyq/W3VBqOJUaMaTWxI3vEJCvagLd3WdBztEapIvGdldbx1yZV yhW7jlXFsWJ32JersFpcg6g73L3LgEPkWkyjTBk7rO+yj57OaPVjMTUeO/Lc2uAfzooF 9msp1UKbYVNYXb72mufxjh9bzWsimKRa8uskV8YqSk4pGqKFyqdYG8AuxIq7Sy58o9MS HuFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=5C7GrvmBCtU7hlvWvFE8nf5I6UlP1swgVhRZJ44sGqI=; b=J9k7FxAQ+JmrcCcrUj8qI5RgdSiPttK6/brJ0HPd71f3r0IAVwZVz50zLb7dQJVpXh ZmzxqV7AWCGjsolgX3ZbaEVR5pWHaZknACxEBOcZwR3HbCzoCok+qtk5iT1v1qpSJgAF iqyKxNzfHp6BIo4IfffdnK9qn0h3I9Ehb/gcZbowComwrirk5OR/l0y3DsaDIN6fUTZe 7e6g9I6Q6Zq1Mezz9eUGphPAGWahq/+1HdN9ntZdBXyLqc6On7e0+1z9laJjz/rsAwDe 2WLWclOTlAHHBjjOAQaGwtO++SUdBvlBJ0AeISIly8aYT/F2lsO+LqxYTzQSVKCGkKTX L8AQ== X-Gm-Message-State: AMke39k8aY8CLv1qZ6d4+Syl/AtXN8QbjXpvB5QfIoIMIev6BltMuO/prDfjrx9JyXemPlHilPFhs3VwEc9tEg== X-Received: by 10.157.27.70 with SMTP id l64mr1399296otl.92.1488296541814; Tue, 28 Feb 2017 07:42:21 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.110.38 with HTTP; Tue, 28 Feb 2017 07:42:01 -0800 (PST) In-Reply-To: References: <6F62EB32-671C-458D-B24B-268491C2DA1E@intel.com> From: Sushil Adhikari Date: Tue, 28 Feb 2017 09:42:01 -0600 Message-ID: To: "Wiles, Keith" Cc: "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] BUG: unable to handle kernel paging request X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2017 15:42:23 -0000 Since printf is not working here I'm using printk, do you mean whether skb_put(skb, len) fails or the memcpy fails( line 169)? I had separated the line 169 in to two one for skb_put and another just memcpy, and it doesn't fail on skb_put so its memory copy that what causing the fail. Since the memory location of data_skv and the location in "BUG: unable to handle kernel paging request at 000007529d212900" matches I thought the data_skv address is not correct or something. On Tue, Feb 28, 2017 at 9:34 AM, Wiles, Keith wrote= : > > > On Feb 28, 2017, at 9:30 AM, Sushil Adhikari > wrote: > > > > its failing at data_kva address because this is where I'm getting the > kernel paging request fail > > BUG: unable to handle kernel paging request at 000007529d212900 > > and this is what my debug shows > > kva data addresses: data_kva 000007529d212900, kva->buff_add > 00007f529d212880, kva->data_off 128, kni->mbuf_va (null), and > kni->mbuf_kva ffff880000000000 > > I was thinking of using GDB to dump memory or use printf to see which one > is failing. > > > > > I'm not sure how to verify that these are normal > > > > On Mon, Feb 27, 2017 at 4:41 PM, Wiles, Keith > wrote: > > > > > On Feb 27, 2017, at 4:22 PM, Sushil Adhikari > wrote: > > > > > > I narrowed it to location where it was failing, its coming from > http://dpdk.org/browse/dpdk-stable/tree/lib/librte_eal/ > linuxapp/kni/kni_net.c?h=3Dv16.07.2 line 169, I am getting the value of l= en > to be 1358 from len=3Dkva->pkt_len; which seems right for ip packet and t= he > memory allocation from line 157 also seems to be working fine. when I pri= nt > the sizeof(*skb) or sizeof(struct sk_buff) its giving me 208, I guess I > dont know whether it should be the size we allocate from line 157, which = is > len + 2 =3D 1360 or its fixed size structure of 208 byte. I would appreci= ate > any insight. > > > Linux version 4.4.20 (sushila@dev03) (gcc version 4.9.2 (crosstool-NG > 1.20.0) ) #1 SMP Fri Feb 24 14:32:28 CST 2017 > > > > Looks like we need to determine which address is failing the skb_put() > or data_kva address. If the address that fails is at the end of the > skb_put() then I would think the len is wrong, meaning we are stepping on > memory just passed a page for the skb. If the address that fails is in th= e > data_kva then the calculations for that address are wrong in line 154. Yo= u > may want to printout the kva->data_off, buf_addr, mbuf_va and mbuf_kva to > verify these values seem normal. The data_off value should be reasonable = (I > guess) meaning with a 2K range. > > > > Also print out the two values skb_put() and data_kva. You can use gdb t= o > example the memory using the dump memory command. (is it x/nn
) = nn > is the read width, but you could leave off the =E2=80=98/nn=E2=80=99 for = the default. > > > > > > > > Thank you > > > Sushil > > > > > > On Sat, Feb 25, 2017 at 10:31 AM, Wiles, Keith > wrote: > > > > > > > On Feb 24, 2017, at 8:07 AM, Sushil Adhikari > wrote: > > > > > > > > Resending because of unsupported email content type > > > > > > > > > > > > yes hanging is the better word I guess, ctrl + c is not working to > actually stop the program. I also had display connected to the target > manchine and I have attached a picture that shows the messages in that > display that is where I saw "BUG:Unable to handle kernel paging request a= t > xxxxxx", which made me think that the program is in bad state. > > > > > > Sorry, I do not see why you are getting this message. All I can > suggest is to use GDB and see if you can determine why the message is > happening. > > > > > > > > > > > info thread in gdb shows one thread running > > > > Id Target Id Frame > > > > * 1 LWP 843 "dpdkKni" 0x000000000044eaee in rte_kni_tx_burst () > > > > > > > > On Thu, Feb 23, 2017 at 5:41 PM, Wiles, Keith > wrote: > > > > > > > > > On Feb 23, 2017, at 2:38 PM, Sushil Adhikari > wrote: > > > > > > > > > > While trying to run dpdk Kni application I ran in to a problem, > with > > > > > following error message > > > > > BUG: unable to handle kernel paging request at 000007ffe2b92780 > > > > > > > > > > To run the application I first unbinded the ports from kernel > module and > > > > > binded them to igb_uio > > > > >> echo 0000:05:00.1 > /sys/bus/pci/drivers/ixgbe/unbind > > > > >> echo 0000:05:00.0 > /sys/bus/pci/drivers/ixgbe/unbind > > > > >> echo 0x8086 0x1528 > /sys/bus/pci/drivers/igb_uio/new_id > > > > > > > > > > I ran the application using gdb as > > > > > > > > > > [~]$ /root/gdb dpdkKni > > > > > GNU gdb (crosstool-NG 1.20.0) 7.8 > > > > > Copyright (C) 2014 Free Software Foundation, Inc. > > > > > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html > > > > >> > > > > > This is free software: you are free to change and redistribute it= . > > > > > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > > > > > and "show warranty" for details. > > > > > This GDB was configured as "x86_64-unknown-linux-gnu". > > > > > Type "show configuration" for configuration details. > > > > > For bug reporting instructions, please see: > > > > > . > > > > > Find the GDB manual and other documentation resources online at: > > > > > . > > > > > For help, type "help". > > > > > Type "apropos word" to search for commands related to "word"... > > > > > Reading symbols from dpdkKni...(no debugging symbols found)...don= e. > > > > > (gdb) Run dpdkKni -c 0x0f -n 4 -- -P -p 0x3 > --config=3D"(0,0,1),(1,2,3)" > > > > > Starting program: /root/dpdkKni dpdkKni -c 0x0f -n 4 -- -P -p 0x3 > > > > > --config=3D"(0,0,1),(1,2,3)" > > > > > warning: Could not load shared library symbols for linux-vdso.so.= 1. > > > > > Do you need "set solib-search-path" or "set sysroot"? > > > > > warning: Unable to find libthread_db matching inferior's thread > library, > > > > > thread debugging will not be available. > > > > > EAL: Detected 4 lcore(s) > > > > > EAL: Probing VFIO support... > > > > > EAL: PCI device 0000:05:00.0 on NUMA socket -1 > > > > > EAL: probe driver: 8086:1528 net_ixgbe > > > > > EAL: PCI device 0000:05:00.1 on NUMA socket -1 > > > > > EAL: probe driver: 8086:1528 net_ixgbe > > > > > Address of pktmbuf_pool 0x7ffff5a7dec0 > > > > > APP: Initialising port 0 ... > > > > > KNI: pci: 05:00:00 8086:1528 > > > > > kni created for port 0 with kni[i] address 0x7fff75638280 with i = 0 > > > > > APP: Initialising port 1 ... > > > > > KNI: pci: 05:00:01 8086:1528 > > > > > kni created for port 1 with kni[i] address 0x7fff75629e00 with i = 0 > > > > > APP: Lcore 1 is writing to port 0 > > > > > APP: Lcore 2 is reading from port 1 > > > > > APP: Lcore 3 is writing to port 1 > > > > > APP: Lcore 0 is reading from port 0 > > > > > ^C > > > > > Program received signal SIGINT, Interrupt. > > > > > > > > The program did not crash or get a segfault, but you hit control-c > which stopped the application. When you ran the application you started 4 > threads and this is why it would appear in different places when stopped. > > > > > > > > If the application is hanging then you can use control-C and then d= o > =E2=80=98info threads=E2=80=99 command to see the location of all threads= . You can use the > =E2=80=98thread X=E2=80=99 command to switch between threads. Please chec= k the command > usage here I am going from memory. > > > > > > > > I am not sure if the application has a -i option to get a command > line if so that maybe useful to enable, check the application to see if i= t > used cmdline feature. > > > > > > > > It maybe the application just sits running and you have to use othe= r > tools or apps to send traffic on the KNI application, sorry I have not > really used the KNI example. > > > > > > > > > 0x000000000044e916 in rte_kni_tx_burst () > > > > > (gdb) backtrace > > > > > #0 0x000000000044e916 in rte_kni_tx_burst () > > > > > #1 0x0000000000619758 in main_loop(void*) () > > > > > #2 0x0000000000431183 in rte_eal_mp_remote_launch () > > > > > #3 0x000000000040d312 in main () > > > > > > > > > > (this is where the program crashes) > > > > > > > > > > I tried to trace the crash with gdb(I am new to gdb) > > > > > > > > > > and when I do the backtrace it ends up in different functions eac= h > time: > > > > > this time it gave me rte_kni_tx_burst() > > > > > > > > > > I'm running latest dpdk version 17.02 and linux kernel is > > > > > Linux version 4.4.20 (tcuser@cibuild08) (gcc version 4.9.2 > (crosstool-NG > > > > > 1.20.0) ) #1 SMP > > > > > > > > > > I would appreciate any suggestion or insight regarding this issue= . > > > > > > > > Regards, > > > > Keith > > > > > > > > > > > > > > > > > > Regards, > > > Keith > > > > > > > > > > Regards, > > Keith > > > > > > Regards, > Keith > >