From: Stefan Baranoff <sbaranoff@gmail.com> To: dev@dpdk.org Subject: [dpdk-dev] Random mbuf corruption Date: Fri, 20 Jun 2014 07:20:52 -0400 Message-ID: <CAHzKxpZUOVKbCYTb66D8cQbm0ceSt7rfYo6VU3f2qhi2ZBvytQ@mail.gmail.com> (raw) In-Reply-To: <CAHzKxpaNvZkH9h0kqYJd8VoYEXqBUfhSX9V_zUro2oX_-ioAAw@mail.gmail.com> All, We are seeing 'random' memory corruption in mbufs coming from the ixgbe UIO driver and I am looking for some pointers on debugging it. Our software was running flawlessly for weeks at a time on our old Westmere systems (CentOS 6.4) but since moving to a new Sandy Bridge v2 server (also CentOS 6.4) it runs for 1-2 minutes and then at least one mbuf is overwritten with arbitrary data (pointers/lengths/RSS value/num segs/etc. are all ridiculous). Both servers are using the 82599EB chipset (x520) and the DPDK version (1.6.0r2) is identical. We recently also tested on a third server running RHEL 6.4 with the same hardware as the failing Sandy Bridge based system and it is fine (days of runtime no failures). Running all of this in GDB with 'record' enabled and setting a watchpoint on the address which contains the corrupted data and executing a 'reverse-continue' never hits the watchpoint [GDB newbie here -- assuming 'watch *(uint64_t*)0x7FB.....' should work]. My first thought was memory corruption but the BIOS memcheck on the ECC RAM shows no issues. Also looking at mbuf->pkt.data, as an example, the corrupt value was the same 6/12 trials but I could not find that value elsewhere in the processes memory. This doesn't seem "random" and points to a software bug but I cannot for the life of me get GDB to tell me where the program is when that memory is written to. Incidentally trying this with the PCAP driver and --no-huge to run valgrind shows no memory access errors/uninitialized values/etc. Thoughts? Pointers? Ways to rule in/out hardware other than going 1 by 1 removing each of the 24 DIMMs? Thanks so much in advance! Stefan
next parent reply other threads:[~2014-06-20 11:20 UTC|newest] Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top [not found] <CAHzKxpaxCbt9d+njdBBpwSy069zLfsOvQ5Dx0CzXLNVMKQ9AaQ@mail.gmail.com> [not found] ` <CAHzKxpaNvZkH9h0kqYJd8VoYEXqBUfhSX9V_zUro2oX_-ioAAw@mail.gmail.com> 2014-06-20 11:20 ` Stefan Baranoff [this message] 2014-06-20 13:59 ` Paul Barrette 2014-06-23 21:43 ` Stefan Baranoff 2014-06-24 8:05 ` Gray, Mark D 2014-06-24 10:48 ` Neil Horman 2014-06-24 11:01 ` Olivier MATZ 2014-06-25 1:31 ` Stefan Baranoff
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAHzKxpZUOVKbCYTb66D8cQbm0ceSt7rfYo6VU3f2qhi2ZBvytQ@mail.gmail.com \ --to=sbaranoff@gmail.com \ --cc=dev@dpdk.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
DPDK patches and discussions This inbox may be cloned and mirrored by anyone: git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \ dev@dpdk.org public-inbox-index dev Example config snippet for mirrors. Newsgroup available over NNTP: nntp://inbox.dpdk.org/inbox.dpdk.dev AGPL code for this site: git clone https://public-inbox.org/public-inbox.git