From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.ndsl.kaist.edu (www.ndsl.kaist.ac.kr [143.248.57.3]) by dpdk.org (Postfix) with ESMTP id 66B449219 for ; Thu, 26 Nov 2015 05:46:53 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mail.ndsl.kaist.edu (Postfix) with ESMTP id D8713C0977 for ; Thu, 26 Nov 2015 13:46:51 +0900 (KST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ndsl.kaist.edu; h=content-transfer-encoding:content-type:content-type :in-reply-to:mime-version:user-agent:date:date:message-id:from :from:references:to:subject:subject; s=dkim; t=1448513207; x= 1449377207; bh=8xEvsraD4bOmnyqpAXp2c6uryJz8VFzB8MJLNrDPnW8=; b=g UskFLR5SZOyVH58TgJaupM3+DvNDQaGgEz2shwfwARHdc/gdWWq4+sWFX+bO2Zmz 7ca1NmqJQy4Yvd5ZUKM1v9YhosS6FdqcW5pEnV9cfBpdE7fQ5o29n9w6Ti6XMCW5 c9SLSxfuQg5jvdIuGXC83e0Vo5W19qn0WsENt/yj1Q= X-Virus-Scanned: Debian amavisd-new at ndsl.kaist.ac.kr Received: from mail.ndsl.kaist.edu ([127.0.0.1]) by localhost (ndsl.kaist.ac.kr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id epznV3qRBn6B for ; Thu, 26 Nov 2015 13:46:47 +0900 (KST) Received: from [10.0.2.15] (ndsl-pc2.kaist.ac.kr [143.248.129.22]) by mail.ndsl.kaist.edu (Postfix) with ESMTPSA id 8AE36C022D for ; Thu, 26 Nov 2015 13:46:46 +0900 (KST) To: dev@dpdk.org References: <56554B08.3040400@ndsl.kaist.edu> <49956413.am4JoMJyVU@xps13> <20151125120239.GA23268@bricha3-MOBL3> <5690109.niDVrFKdOE@xps13> <5655BB2C.4090806@intel.com> <2601191342CEEE43887BDE71AB97725836ACD0DB@irsmsx105.ger.corp.intel.com> From: Younghwan Go Message-ID: <56568EC7.6010905@ndsl.kaist.edu> Date: Thu, 26 Nov 2015 13:47:03 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <2601191342CEEE43887BDE71AB97725836ACD0DB@irsmsx105.ger.corp.intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] no hugepage with UIO poll-mode driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Nov 2015 04:46:54 -0000 Hello, Thank you all for helping us understand on issues with no hugepage option. As Konstantin mentioned at the end, I tried using VFIO module instead of IGB UIO module. I enabled all necessary parameters (IOMMU, virtualization, vfio-pci, VFIO permission) and ran my code with no hugepage option. At first, it seemed to receive packets fine, but after a while, it stopped receiving packets. I could temporarily remove this issue by not calling rte_eth_tx_burst(). Also, when I looked at the received packets, they all contained 0s instead of actual data. Was there anything that I missed in running with VFIO? I'm curious if no hugepage with no hugepage option was confirmed to run with VFIO. Thank you, Younghwan 2015-11-25 오후 11:12에 Ananyev, Konstantin 이(가) 쓴 글: > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Sergio Gonzalez Monroy >> Sent: Wednesday, November 25, 2015 1:44 PM >> To: Thomas Monjalon >> Cc: dev@dpdk.org >> Subject: Re: [dpdk-dev] no hugepage with UIO poll-mode driver >> >> On 25/11/2015 13:22, Thomas Monjalon wrote: >>> 2015-11-25 12:02, Bruce Richardson: >>>> On Wed, Nov 25, 2015 at 12:03:05PM +0100, Thomas Monjalon wrote: >>>>> 2015-11-25 11:00, Bruce Richardson: >>>>>> On Wed, Nov 25, 2015 at 11:23:57AM +0100, Thomas Monjalon wrote: >>>>>>> 2015-11-25 10:08, Bruce Richardson: >>>>>>>> On Wed, Nov 25, 2015 at 03:39:17PM +0900, Younghwan Go wrote: >>>>>>>>> Hi Jianfeng, >>>>>>>>> >>>>>>>>> Thanks for the email. rte mempool was successfully created without any >>>>>>>>> error. Now the next problem is that rte_eth_rx_burst() is always returning 0 >>>>>>>>> as if there was no packet to receive... Do you have any suggestion on what >>>>>>>>> might be causing this issue? In the meantime, I will be digging through >>>>>>>>> ixgbe driver code to see what's going on. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Younghwan >>>>>>>>> >>>>>>>> The problem is that with --no-huge we don't have the physical address of the memory >>>>>>>> to write to the network card. That's what it's marked as for testing only. >>>>>>> Even with rte_mem_virt2phy() + rte_mem_lock_page() ? >>>>>>> >>>>>> With no-huge, we just set up a single memory segment at startup and set its >>>>>> "physaddr" to be the virtual address. >>>>>> >>>>>> /* hugetlbfs can be disabled */ >>>>>> if (internal_config.no_hugetlbfs) { >>>>>> addr = mmap(NULL, internal_config.memory, PROT_READ | PROT_WRITE, >>>>>> MAP_PRIVATE | MAP_ANONYMOUS, 0, 0); >>>>>> if (addr == MAP_FAILED) { >>>>>> RTE_LOG(ERR, EAL, "%s: mmap() failed: %s\n", __func__, >>>>>> strerror(errno)); >>>>>> return -1; >>>>>> } >>>>>> mcfg->memseg[0].phys_addr = (phys_addr_t)(uintptr_t)addr; >>>>> rte_mem_virt2phy() does not use memseg.phys_addr but /proc/self/pagemap: >>>>> >>>>> /* >>>>> * the pfn (page frame number) are bits 0-54 (see >>>>> * pagemap.txt in linux Documentation) >>>>> */ >>>>> physaddr = ((page & 0x7fffffffffffffULL) * page_size) >>>>> + ((unsigned long)virtaddr % page_size); >>>>> >>>> Yes, you are right. I was not aware that that function was used as part of the >>>> mempool init, but now I see that "rte_mempool_virt2phy()" does indeed call that >>>> function if hugepages are disabled, so my bad. >>> Do you think we could move --no-huge in the main section (not only for testing)? >> Hi, >> >> I think the main issue is going to be the HW descriptors queues. >> AFAIK drivers now call rte_eth_dma_zone_reserve, which is basically a >> wrapper around >> rte_memzone_reserve, to get a chunk of phys memory, and in the case of >> --no-huge is >> not going to be really phys contiguous. >> >> Ideally we would move and expand the functionality of dma_zone reserve >> API to the EAL, >> so we could detect what page size we have and set the boundary for such >> page size. >> dma_zone_reserve does something similar to work on Xen target by >> reserving memzones >> on 2MB boundary. > With xen we have a special kernel driver that allocates physically continuous > chunks of memory for us. > So we can guarantee that each such chunk would be at least 2MB long. > That's enough to allocate HW rings (max HW ring size for let say ixgbe is ~64KB). > Here there is absolutely no guarantee that memory allocated by kernel will be memory continuous. > Of course you can search though all pages that you allocated and most likely you'll find a continuous > chunk big enough for that. > Another problem - mbufs. > You need to be sure that each mbuf doesn't cross page boundary > (in case next page is not adjacent to current one). > So you'll probably need to use rte_mempool_xmem_create() to allocate mbufs from no hugepages. > BTW, as I remember with vfio in place you should be able to do IO with no-hugepages options, no? > As it relies on vfio ability to setup IOMMU tables for you. > Konstantin > >> Sergio