DPDK patches and discussions
 help / color / mirror / Atom feed
From: Neil Horman <nhorman@tuxdriver.com>
To: Stefan Baranoff <sbaranoff@gmail.com>
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Random mbuf corruption
Date: Tue, 24 Jun 2014 06:48:59 -0400
Message-ID: <20140624104859.GA19229@hmsreliant.think-freely.org> (raw)
In-Reply-To: <CAHzKxpYaUhR5ti2EDZfj7jeu8pWxhnmWM+e2D20k01NHa_u85w@mail.gmail.com>

On Mon, Jun 23, 2014 at 05:43:21PM -0400, Stefan Baranoff wrote:
> Paul,
> 
> Thanks for the advice; we ran memtest as well as the Dell complete system
> diagnostic and neither found an issue. The plot thickens, though!
> 
> Our admins messed up our kickstart labels and what I *thought* was CentOS
> 6.4 was actually RHEL 6.4 and the problem seems to be following the CentOS
> 6.4 installations -- the current configuration of success/failure is:
>   1 server - Westmere - RHEL 6.4 -- works
>   1 server - Sandy Bridge - RHEL 6.4 -- works
>   2 servers - Sandy Bridge - CentOS 6.4 -- fails
> 
there were several memory corruptors fixed in between RHEL 6.3 and RHEL 6.5.
Its possible that CentOS didn't get one of those patches, if it went out in a
zstream or something.  Is there an older version of RHEL that recreates the
problem for you?  If so, I can provide a list of bugs/fixes that may be related,
which you could cross check against centos for inclusion.

> Given that the hardware seems otherwise stable/checks out I'm trying to
> figure out how to determine if this is:
>   a) our software has a bug
>   b) a kernel/hugetlbfs bug
>   c) a  DPDK 1.6.0r2 bug
> 
> I have seen similar issues where calling rte_eal_init too late in a process
> also causes similar issues (things like calling 'free' on memory that was
> allocated with 'malloc' before 'rte_eal_init' is called fails/results in
> segfault in libc) which seems odd to me but in this case we are calling
> rte_eal_init as the first thing we do in main().
> 
Sounds like it might be time to add in some poisoning options to the mbuf
allocator.

Neil

> 
> Thanks,
> Stefan
> 
> 
> On Fri, Jun 20, 2014 at 9:59 AM, Paul Barrette <paul.barrette@windriver.com>
> wrote:
> 
> >
> > On 06/20/2014 07:20 AM, Stefan Baranoff wrote:
> >
> >> All,
> >>
> >> We are seeing 'random' memory corruption in mbufs coming from the ixgbe
> >> UIO
> >> driver and I am looking for some pointers on debugging it. Our software
> >> was
> >> running flawlessly for weeks at a time on our old Westmere systems (CentOS
> >> 6.4) but since moving to a new Sandy Bridge v2 server (also CentOS 6.4) it
> >> runs for 1-2 minutes and then at least one mbuf is overwritten with
> >> arbitrary data (pointers/lengths/RSS value/num segs/etc. are all
> >> ridiculous). Both servers are using the 82599EB chipset (x520) and the
> >> DPDK
> >> version (1.6.0r2) is identical. We recently also tested on a third server
> >> running RHEL 6.4 with the same hardware as the failing Sandy Bridge based
> >> system and it is fine (days of runtime no failures).
> >>
> >> Running all of this in GDB with 'record' enabled and setting a watchpoint
> >> on the address which contains the corrupted data and executing a
> >> 'reverse-continue' never hits the watchpoint [GDB newbie here -- assuming
> >> 'watch *(uint64_t*)0x7FB.....' should work]. My first thought was memory
> >> corruption but the BIOS memcheck on the ECC RAM shows no issues.
> >>
> >> Also looking at mbuf->pkt.data, as an example, the corrupt value was the
> >> same 6/12 trials but I could not find that value elsewhere in the
> >> processes
> >> memory. This doesn't seem "random" and points to a software bug but I
> >> cannot for the life of me get GDB to tell me where the program is when
> >> that
> >> memory is written to. Incidentally trying this with the PCAP driver and
> >> --no-huge to run valgrind shows no memory access errors/uninitialized
> >> values/etc.
> >>
> >> Thoughts? Pointers? Ways to rule in/out hardware other than going 1 by 1
> >> removing each of the 24 DIMMs?
> >>
> >> Thanks so much in advance!
> >> Stefan
> >>
> > Run memtest to rule out bad ram.
> >
> > Pb
> >
> 

  parent reply	other threads:[~2014-06-24 10:48 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CAHzKxpaxCbt9d+njdBBpwSy069zLfsOvQ5Dx0CzXLNVMKQ9AaQ@mail.gmail.com>
     [not found] ` <CAHzKxpaNvZkH9h0kqYJd8VoYEXqBUfhSX9V_zUro2oX_-ioAAw@mail.gmail.com>
2014-06-20 11:20   ` Stefan Baranoff
2014-06-20 13:59     ` Paul Barrette
2014-06-23 21:43       ` Stefan Baranoff
2014-06-24  8:05         ` Gray, Mark D
2014-06-24 10:48         ` Neil Horman [this message]
2014-06-24 11:01           ` Olivier MATZ
2014-06-25  1:31             ` Stefan Baranoff

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20140624104859.GA19229@hmsreliant.think-freely.org \
    --to=nhorman@tuxdriver.com \
    --cc=dev@dpdk.org \
    --cc=sbaranoff@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ https://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git