From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f68.google.com (mail-oi0-f68.google.com [209.85.218.68]) by dpdk.org (Postfix) with ESMTP id 2DFD75A3E for ; Thu, 23 Feb 2017 18:03:20 +0100 (CET) Received: by mail-oi0-f68.google.com with SMTP id z13so1743553oig.2 for ; Thu, 23 Feb 2017 09:03:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Ntsmy0jDLs+O+miCqwXQN4ggf6Nor9DZRJ4v/AuUTG4=; b=NV5cz801NV27ySi3tAUmG2zHH2huMtliJUIwSIcvFYERmPybliJYsmxYQaYuz9MyxP ZLxpeP8fH+my8cgC8sFZYyKzFnQ+wIJbzdKHTExs7R+R8vOrYXweG6zGEmLrU+Kjl0Kb CRQ61cLR2eAUT446WSwKI/JWf3bSicNIx//PMzadhJaIVdGbIXqFI7DY5wGFHMn5Kkn/ Bk+K5XIYFK2jDlkrK6ToZAMV20wDuFYqmyyrQS64TmugB0foHaUwKNUD0AfS+cQH0VW0 MoWv5TlK+qKYtV4RecFoFxwX6kJMRiKidKeJM4QVUz89ZLhDpWecIpKdIgSvahIGnGsq b63g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Ntsmy0jDLs+O+miCqwXQN4ggf6Nor9DZRJ4v/AuUTG4=; b=o8iiRjpfvM2QKVu38gfiGX+4OgrKNiBVGc8QAdvuOEeuOLHcgJS+MF9h5hFrTb2mLk UzHNZR6+cDSk1oGTH8MsjeggkUPngIlseqvKkYHS6CIIbY+vwQ8/CSgddUyBvIGaKSvi 1HkZEQ159M58RNHa/9MLmf06l5xNs8dKLLc1tz/3sSPcynPtYLj67nBrzD4bwW6T3xf8 Jp1fjE6ZhmchEoiQsDO5NUCTVA3ryWhE7EipKARMXosolFGO2g7IjpKdOcjNMhEDLE16 4TWg/jlwVAPlOoj3DvKRdWUjU93vi/JpaXovdtQpUN5QUYh+EqpI6BweW0OR5qkYP8nH QgQw== X-Gm-Message-State: AMke39kiV/rjBqgOiui9DepWDuo5bNQ6F+f/MWxUJ9Prr8qDhNC5Bm366ZzQFFYS8QqFiO6hUpITuX/xvhboyQ== X-Received: by 10.202.55.197 with SMTP id e188mr23033803oia.12.1487869400145; Thu, 23 Feb 2017 09:03:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.110.38 with HTTP; Thu, 23 Feb 2017 09:02:59 -0800 (PST) In-Reply-To: References: <18013811-F9E1-42C6-A87B-D087018AF0B9@intel.com> <6649EDAE-E9E4-4C49-855C-56E37BD4FA88@intel.com> <986FF8E4-1D37-4275-B9E3-8C7E19996461@intel.com> From: Sushil Adhikari Date: Thu, 23 Feb 2017 11:02:59 -0600 Message-ID: To: Sergio Gonzalez Monroy Cc: "Wiles, Keith" , "users@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Larger number of hugepages causes bus error. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Feb 2017 17:03:21 -0000 Thank you Keith and Monroy, with your help I was able to track down the problem, My var/run was too small to hold the hugepage information so when I increased its size, it worked. Thank you so much. On Thu, Feb 23, 2017 at 10:35 AM, Sergio Gonzalez Monroy < sergio.gonzalez.monroy@intel.com> wrote: > As Keith suggested, gdb is probably your best bet now. > You could also do 'strace' to see if something shows up there. > > If you are running as root, the application is opening a file in /var/run > to store some hugepage information, then it memsets to 0. > > What distro and kernel are you running on? > > > > On 23/02/2017 16:19, Sushil Adhikari wrote: > >> I didn't understand what you mean by hugepage value, if you mean number = of >> hugepages here's what it looks like >> [~]$ grep -ri hugepages /proc/meminfo >> AnonHugePages: 0 kB >> HugePages_Total: 512 >> HugePages_Free: 512 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> And the linux version is 4.4.20. >> >> On Thu, Feb 23, 2017 at 9:17 AM, Wiles, Keith >> wrote: >> >> On Feb 22, 2017, at 7:18 PM, Sushil Adhikari >>>> >>> wrote: >>> >>>> Thank you Keith for the response, >>>> >>>> Yes it should be line 1142 not 1405, I was using 16.11 and now I'm usi= ng >>>> >>> 17.02 and still getting the same error. >>> >>> Not sure what to say here, it looks like some type of system >>> configuration >>> issue as I do not see it on my machine. >>> >>> Can you tell if the hugepage has a value and is it sane? The next thing >>> is >>> to see where in that memory is it failing start, end or middle someplac= e. >>> Use GDB and compile the code with =E2=80=98make install >>> T=3Dx86_64-native-lunixapp-gcc EXTRA_CFLAGS=3D=E2=80=9C-g -O0=E2=80=9D'= then set a break >>> point >>> on =E2=80=98b eal_memory.c:1142=E2=80=99 and inspect the memory pointer= hugepage. I do >>> not >>> think it is overrun error meaning the size for memset is different then >>> what was allocated and just stepping off the end. >>> >>> Also you did not tell me the linux version you are using? >>> >>> On Wed, Feb 22, 2017 at 8:46 PM, Wiles, Keith >>>> >>> wrote: >>> >>>> On Feb 22, 2017, at 6:43 PM, Wiles, Keith >>>>> >>>> wrote: >>> >>>> On Feb 22, 2017, at 6:30 PM, Sushil Adhikari >>>>>> >>>>> wrote: >>> >>>> I used the basic command line option "dpdkTimer -c 0xf -n 4" >>>>>> And to update on my findings so far I have narrowed down to this >>>>>> >>>>> line(1405) >>> >>>> memset(hugepage, 0, nr_hugefiles * sizeof(struct hugepage_file)); >>>>>> of function rte_eal_hugepage_init() in file >>>>>> >>>>> dpdk\lib\librte_eal\linuxapp\eal\eal_memory.c >>> >>>> What version of DPDK are you using? I was looking at the file at 1405 >>>>> >>>> and I do not see a memset() call. >>> >>>> I found the memset call at 1142 in my 17.05-rc0 code. Please try the >>>> >>> latest version and see if you get the same problem. >>> >>>> Yes I have the hugepages of size 2MB(2048) and when I calculate the >>>>>> >>>>> memory this memset function is trying to set, it comes out to >>> 512(nr_hugefiles) * 4144 ( sizeof(struct hugepage_file) ) =3D 2121728 w= hich >>> larger than 2MB, so my doubt is that the hugepages I have >>> allocated(512*2MB) is not contiguous 1GB memory its trying to access >>> memory >>> thats not part of hugepage, is that a possibility, even though I am >>> setting >>> up hugepages during boot time by providing it through kernel option. >>> >>>> >>>>>> On Wed, Feb 22, 2017 at 8:05 PM, Wiles, Keith >>>>>> >>>>> wrote: >>> >>>> On Feb 22, 2017, at 3:05 PM, Sushil Adhikari >>>>>>> >>>>>> wrote: >>> >>>> Hi, >>>>>>> >>>>>>> I was trying to run dpdk timer app by setting 512 2MB hugepages but >>>>>>> >>>>>> the >>> >>>> application crashed with following error >>>>>>> EAL: Detected 4 lcore(s) >>>>>>> EAL: Probing VFIO support... >>>>>>> Bus error (core dumped) >>>>>>> >>>>>>> If I reduce the number of hugepages to 256 it works fine. I >>>>>>> >>>>>> wondering what >>> >>>> could be the problem here. Here's my cpu info >>>>>>> >>>>>> I normally run with 2048 x 2 or 2048 per socket on my machine. What >>>>>> >>>>> is the command line you are using to start the application? >>> >>>> processor : 0 >>>>>>> vendor_id : GenuineIntel >>>>>>> cpu family : 6 >>>>>>> model : 26 >>>>>>> model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz >>>>>>> stepping : 5 >>>>>>> microcode : 0x11 >>>>>>> cpu MHz : 2794.000 >>>>>>> cache size : 8192 KB >>>>>>> physical id : 0 >>>>>>> siblings : 4 >>>>>>> core id : 0 >>>>>>> cpu cores : 4 >>>>>>> apicid : 0 >>>>>>> initial apicid : 0 >>>>>>> fpu : yes >>>>>>> fpu_exception : yes >>>>>>> cpuid level : 11 >>>>>>> wp : yes >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr >>>>>>> >>>>>> pge mca >>> >>>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe >>>>>>> >>>>>> syscall nx >>> >>>> rdtscp lm constant_tsc arch_ >>>>>>> perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni >>>>>>> >>>>>> dtes64 >>> >>>> monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt >>>>>>> lahf_lm ida dtherm tpr_shadow vnm >>>>>>> i flexpriority ept vpid >>>>>>> bugs : >>>>>>> bogomips : 5600.00 >>>>>>> clflush size : 64 >>>>>>> cache_alignment : 64 >>>>>>> address sizes : 36 bits physical, 48 bits virtual >>>>>>> power management: >>>>>>> >>>>>>> processor : 1 >>>>>>> vendor_id : GenuineIntel >>>>>>> cpu family : 6 >>>>>>> model : 26 >>>>>>> model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz >>>>>>> stepping : 5 >>>>>>> microcode : 0x11 >>>>>>> cpu MHz : 2794.000 >>>>>>> cache size : 8192 KB >>>>>>> physical id : 0 >>>>>>> siblings : 4 >>>>>>> core id : 1 >>>>>>> cpu cores : 4 >>>>>>> apicid : 2 >>>>>>> initial apicid : 2 >>>>>>> fpu : yes >>>>>>> fpu_exception : yes >>>>>>> cpuid level : 11 >>>>>>> wp : yes >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr >>>>>>> >>>>>> pge mca >>> >>>> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe >>>>>>> >>>>>> syscall nx >>> >>>> rdtscp lm constant_tsc arch_ >>>>>>> perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni >>>>>>> >>>>>> dtes64 >>> >>>> monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt >>>>>>> lahf_lm ida dtherm tpr_shadow vnm >>>>>>> i flexpriority ept vpid >>>>>>> bugs : >>>>>>> bogomips : 5600.00 >>>>>>> clflush size : 64 >>>>>>> cache_alignment : 64 >>>>>>> address sizes : 36 bits physical, 48 bits virtual >>>>>>> power management:...... >>>>>>> >>>>>>> And Here's my meminfo >>>>>>> >>>>>>> MemTotal: 24679608 kB >>>>>>> MemFree: 24014156 kB >>>>>>> MemAvailable: 23950600 kB >>>>>>> Buffers: 3540 kB >>>>>>> Cached: 31436 kB >>>>>>> SwapCached: 0 kB >>>>>>> Active: 21980 kB >>>>>>> Inactive: 22256 kB >>>>>>> Active(anon): 10760 kB >>>>>>> Inactive(anon): 2940 kB >>>>>>> Active(file): 11220 kB >>>>>>> Inactive(file): 19316 kB >>>>>>> Unevictable: 0 kB >>>>>>> Mlocked: 0 kB >>>>>>> SwapTotal: 0 kB >>>>>>> SwapFree: 0 kB >>>>>>> Dirty: 32 kB >>>>>>> Writeback: 0 kB >>>>>>> AnonPages: 9252 kB >>>>>>> Mapped: 11912 kB >>>>>>> Shmem: 4448 kB >>>>>>> Slab: 27712 kB >>>>>>> SReclaimable: 11276 kB >>>>>>> SUnreclaim: 16436 kB >>>>>>> KernelStack: 2672 kB >>>>>>> PageTables: 1000 kB >>>>>>> NFS_Unstable: 0 kB >>>>>>> Bounce: 0 kB >>>>>>> WritebackTmp: 0 kB >>>>>>> CommitLimit: 12077660 kB >>>>>>> Committed_AS: 137792 kB >>>>>>> VmallocTotal: 34359738367 kB >>>>>>> VmallocUsed: 0 kB >>>>>>> VmallocChunk: 0 kB >>>>>>> HardwareCorrupted: 0 kB >>>>>>> AnonHugePages: 2048 kB >>>>>>> CmaTotal: 0 kB >>>>>>> CmaFree: 0 kB >>>>>>> HugePages_Total: 256 >>>>>>> HugePages_Free: 0 >>>>>>> HugePages_Rsvd: 0 >>>>>>> HugePages_Surp: 0 >>>>>>> Hugepagesize: 2048 kB >>>>>>> DirectMap4k: 22000 kB >>>>>>> DirectMap2M: 25133056 kB >>>>>>> >>>>>> Regards, >>>>>> Keith >>>>>> >>>>>> >>>>>> Regards, >>>>> Keith >>>>> >>>> Regards, >>>> Keith >>>> >>>> >>>> Regards, >>> Keith >>> >>> >>> >