From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vk0-f42.google.com (mail-vk0-f42.google.com [209.85.213.42]) by dpdk.org (Postfix) with ESMTP id B15655681 for ; Fri, 18 Mar 2016 18:24:32 +0100 (CET) Received: by mail-vk0-f42.google.com with SMTP id e185so149618269vkb.1 for ; Fri, 18 Mar 2016 10:24:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=MOhUjbswI1/yxXOQ8RffFaOAzAyETi0ITC8vWof74IA=; b=YPZAxhSpxPiXsccuc7zY+Ou7s8sG99TQADewDwxROY34ZHwfTTP6EBL/35Tps6FEe0 nyHlMCOQTHN/bpyZ2SDus0rup9yWwlAS6oWiXh94anKq4BH5bbGd6TCX/OTs+X35nAHl dSddgubW/56GW2gEwT0nOuhCNdIM5bqEEW4iQFUGWL5X4zcG4OU+DlFPdek0AuHjcC/p UDK/G7F1jNEN8sGGrNbAEFPiVLGlJgqpgaRWQYuxQxV2/bpoEQ1sxiHwhzinNatA30MJ uIslrS2LFCdN6BB9aEc6rhNdAQNFfOWfp+kRgJa+nl3CxfHIw59s5LCdA2rOpP38kXLF yxKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=MOhUjbswI1/yxXOQ8RffFaOAzAyETi0ITC8vWof74IA=; b=JTEm+oieg54uWW1f+W+KPgwc/Su+d0uITNZSoTgV75+7mrebdbuJbkuu1Snif4KsdR 28eo4pVI5x32hfIxqJ7PEmHyUhqlhJoRQjRS+FryUVJayW0xGJ2c36yA35+7l8XKPpFk pjNyHbmUNn6UxXX7Mr6yHFJN8VCcPWOKSLHsIE/pkIcZfttXaCZ/H0jWvGLedC95YSLh n2bgKVhCJhiftBT2tl2c37W8+K6IE4aaTDvnRO0s+91XKZJI1wk75L7d8Slxv+KKtPJa HLbTcSFQ2NHjxh8XIt3wN5n+Gsktr82vDP5BE68odgJPrCtJ6mK+RdKEH0m+aI6fwpYC AeEA== X-Gm-Message-State: AD7BkJIJLibz1c00QN3Ix5ROVTZ3gUtnSkO8L31uLpn6gJ6N7/A6H8izhy8xmBcJZdaROGfeLbjDKzeJaIxkUQ== MIME-Version: 1.0 X-Received: by 10.31.148.215 with SMTP id w206mr16519689vkd.65.1458321872188; Fri, 18 Mar 2016 10:24:32 -0700 (PDT) Received: by 10.31.64.138 with HTTP; Fri, 18 Mar 2016 10:24:32 -0700 (PDT) In-Reply-To: <56EB6D29.9020907@intel.com> References: <56EB6D29.9020907@intel.com> Date: Fri, 18 Mar 2016 10:24:32 -0700 Message-ID: From: John Wei To: "Tan, Jianfeng" Cc: dev@dpdk.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Fwd: EAL: map_all_hugepages(): mmap failed: Cannot allocate memory X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 17:24:33 -0000 Thanks for the reply. Upon further debugging, I was able to root caused the issue. In the cgroup, in addition to limiting the CPU, I also limited the node where my OVS can allocate the memory (cpuset.mems). I understand that DPDK first grab all the memory, then pick the best memory pages, then release the rest. But this is taking a long time for my case that I started many OVSs on the same host. Each DPDK app will need to wait for previous app to release the memory before next app can proceed. In addition, since I have specified that (through cgroup cpuset.mems) dont get memory from other node, DPDK library may be can skipp grabbing memory from these excluded nodes? Just some thoughts. John On Thu, Mar 17, 2016 at 7:51 PM, Tan, Jianfeng wrote: > > > On 3/18/2016 6:41 AM, John Wei wrote: > > I am setting up OVS inside a Linux container. This OVS is built using DPDK > library. > During the startup of ovs-vswitchd, it core dumped due to fail to mmap. > in eal_memory.c > virtaddr = mmap(vma_addr, hugepage_sz, PROT_READ | PROT_WRITE, > MAP_SHARED, fd, 0); > > This call is made inside a for loop that loops through all the pages and > mmap them. > My server has two cores, and I allocated 8192 2MB pages. > The mmap for the first 4096 pages were successful. It failed when trying to > map 4096th page. > > Can someone help me understand when the mmap for the first 4096 pages were > successful and it failed on 4096th page? > > > In my limited experience, there are some scenario that may lead to such > failure: a. specified an option size when do mount hugetlbfs; b. cgroup > limitation, /sys/fs/cgroup/hugetlb/ name>/hugetlb.2MB.limit_in_bytes; c. open files by ulimit... > > Workaround: as only "--socket-mem 128,128" is needed, you can reduce the > total number of 2M hugepages from 8192 to 512 (or something else). > In addition: this is a case why I sent a patchset: > http://dpdk.org/dev/patchwork/patch/11194/ > > Thanks, > Jianfeng > > > > John > > > > ovs-vswitchd --dpdk -c 0x1 -n 4 -l 1 --file-prefix ct0000- --socket-mem > 128,128 -- unix:$DB_SOCK --pidfile --detach --log-file=ct.log > > > EAL: Detected lcore 23 as core 5 on socket 1 > EAL: Support maximum 128 logical core(s) by configuration. > EAL: Detected 24 lcore(s) > EAL: No free hugepages reported in hugepages-1048576kB > EAL: VFIO modules not all loaded, skip VFIO support... > EAL: Setting up physically contiguous memory... > EAL: map_all_hugepages(): mmap failed: Cannot allocate memory > EAL: Failed to mmap 2 MB hugepages > PANIC in rte_eal_init(): > Cannot init memory > 7: [ovs-vswitchd() [0x411f15]] > 6: [/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff5f6133b15]] > 5: [ovs-vswitchd() [0x4106f9]] > 4: [ovs-vswitchd() [0x66917d]] > 3: [ovs-vswitchd() [0x42b6f5]] > 2: [ovs-vswitchd() [0x40dd8c]] > 1: [ovs-vswitchd() [0x56b3ba]] > Aborted (core dumped) > > >