From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f169.google.com (mail-wr0-f169.google.com [209.85.128.169]) by dpdk.org (Postfix) with ESMTP id 8A4161396 for ; Fri, 9 Jun 2017 10:27:30 +0200 (CEST) Received: by mail-wr0-f169.google.com with SMTP id g76so27208564wrd.1 for ; Fri, 09 Jun 2017 01:27:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=41hgKdmgTZ5dZcdxMJUbBJMZUGQQOuyzICXKjAZSNWU=; b=VYM2y5qROhiYXcZjkS75JDnFFFlFqrRFW/xRA97KOLCIk3hmPobBWNY/PoC7jVZbrY TRMp1pA4yQt0bJrWpXCCTS9K3SEczu61bkWfKLKR9uHRHQqk1X61ZrEIL1vn5s8Ehebz ux0LOa1mSYHDNs1haMBJYO46VRxkQKuWlQMH14uTZ6aR/hq35oP7wkQrjHFYJ6FSqLP6 BtCScY3p1yaesInbdt580S83K1U15Tj9MyELdFqcb/gfb2qsOKPFqlC9gMviGnma004Y C68fUFjuYioLVP2NQ/anDSXHdXpTi2FBIHIVzIGgRfuiBAI4L+rEfT/2f1I4OUPFrAe1 ZVvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=41hgKdmgTZ5dZcdxMJUbBJMZUGQQOuyzICXKjAZSNWU=; b=E+OsSO4mMxll+GqeOwwcfT9qJyXaxU7iPUKwfnglu51UogbqkD7rJwMrxNkogRpziL yu9WdkMZ9Tt1L7OzVTkyqialHNEzo4tl/n033f5+MWHVzN1peGv8o/SFi8fu/YDrCyJ6 4zRjm7mrwpvNwz8D80iAMWL3Q7MDwqVtaK3pc8uBDaen9O1WW+xcoJHFZfQPdWQ0JgbX G5piIEeg2wygtTaX1arvOP2IeSe0QcJ+spWj3yA+8rDPPFqFKTHjBtSHEt3nVurEXY4I 7slsmQO0Go6WyBqiL28de0ts+fMZwt2T1RSODLuhd0cPrn+0wMPMcY5J7komf7qiS2H0 YFRA== X-Gm-Message-State: AODbwcC+AFrCu8gYihwkr0+sKmRIWXhdl2eq0XK1fx/U9LUSJg/IyGpG D6UOzE/T/R8HhPNA X-Received: by 10.223.138.164 with SMTP id y33mr20178564wry.144.1496996850246; Fri, 09 Jun 2017 01:27:30 -0700 (PDT) Received: from platinum (2a01cb0c03c651000226b0fffeed02fc.ipv6.abo.wanadoo.fr. [2a01:cb0c:3c6:5100:226:b0ff:feed:2fc]) by smtp.gmail.com with ESMTPSA id h73sm1845796wma.10.2017.06.09.01.27.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 09 Jun 2017 01:27:30 -0700 (PDT) Date: Fri, 9 Jun 2017 10:27:27 +0200 From: Olivier Matz To: Ilya Matveychikov Cc: dev@dpdk.org, "adrien.mazarguil@6wind.com" , Jan Blunck , Sergio Gonzalez Monroy Message-ID: <20170609102727.0eb7f39d@platinum> In-Reply-To: <1A9D36CE-8B6D-43A7-BE0C-9F232DFDA263@gmail.com> References: <1A9D36CE-8B6D-43A7-BE0C-9F232DFDA263@gmail.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] A (possible) problem with `--no-huge` option X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jun 2017 08:27:30 -0000 Hi Ilya, On Sun, 14 May 2017 14:34:14 +0400, Ilya Matveychikov wrote: > Hi guys, >=20 > I have a problem while running DPDK with `--no-huge` option. It seems tha= t the problem occurs since commit cdc242f260e766bd95a658b5e0686a62ec04f5b0 = and that is the change that affects me: >=20 > + if ((page & 0x7fffffffffffffULL) =3D=3D 0) > + return RTE_BAD_PHYS_ADDR; > + >=20 > What I did is to try to create memory pool using rte_pktmbuf_pool_create(= ). I dig into the issue and found that in my case =E2=80=9Cpage" value is 0= x0080000000000000 which means that the page is not present and =E2=80=9Csof= t-dirty=E2=80=9D (according to kernel=E2=80=99s documentation): >=20 > * Bits 0-54 page frame number (PFN) if present > * Bits 0-4 swap type if swapped > * Bits 5-54 swap offset if swapped > * Bit 55 pte is soft-dirty (see Documentation/vm/soft-dirty.txt) > * Bit 56 page exclusively mapped (since 4.2) > * Bits 57-60 zero > * Bit 61 page is file-page or shared-anon (since 3.5) > * Bit 62 page swapped > * Bit 63 page present >=20 > So, before the change mentioned all =E2=80=9Cworks=E2=80=9D fine and such= pages were not handled. But now the check causes rte_mempool_populate_defa= ult to fail with -EINVAL... > Can anyone familiar with the memory pool allocation helps with the issue? >=20 > Thanks in advice, > Ilya Matveychikov. >=20 I can reproduce the issue: make config T=3Dx86_64-native-linuxapp-gcc make -j32 EXTRA_CFLAGS=3D"-O0 -g" mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_h= ugepages # ok ./build/app/testpmd -l 2,4 --log-level 8 --vdev=3Deth_null0 -- --no-numa = --total-num-mbufs=3D4096 -i --port-topology=3Dchained # fail ./build/app/testpmd --no-huge -l 2,4 --log-level 8 --vdev=3Deth_null0 -- = --no-numa --total-num-mbufs=3D4096 -i --port-topology=3Dchained I confirm that rte_mem_virt2phy() returns RTE_BAD_PHYS_ADDR, which makes rte_mempool_populate_virt() to fail. Reverting cdc242f260e7 ("eal/linux: support running as unprivileged user") fixes the problem. Actually, it makes rte_mem_virt2phy() return 0 instead of RTE_BAD_PHYS_ADDR, which is seen as a valid address. I think querying the physical address when using --no-huge does not make sense because the memory is not locked, and could be swapped. Another strange thing, when using --no-huge, the physical address returned when allocating a memzone is the virtual address. I see several solutions to fix the issue: 1/ Always set physical addresses to RTE_BAD_PHYS_ADDR when started with --no-huge. We consider that the physical address is invalid in that case and must not be used. This impacts rte_mem_virt2phy() and memzone_reserve*() functions. In rte_mempool_populate_virt(), don't expect a physical address if the application is started with --no-huge. 2/ Change rte_mem_virt2phy() to return the virtual address when we ask for the physical address when started with --no-huge. This is wrong, but consistent with what is done in memzones today. In rte_mem_virt2phy(), add at the beginning: if (!rte_eal_has_hugepages()) return (intptr_t)virtaddr; 3/ lock pages in memory by reverting 729f17a932dd ("mem: revert page locking when not using hugepages") This would make the physical address available. As explained in the commit log, this would also break the ability to start dpdk with --no-huge for non-root users. I think 1/ is better. I'm sending a patch in reply to this mail. Ilya, please let me know if it fixes your issue. Regards, Olivier