From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0D2D5A04C0; Fri, 25 Sep 2020 17:15:29 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 79DED1E998; Fri, 25 Sep 2020 17:15:26 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by dpdk.org (Postfix) with ESMTP id 1BEE31E89E for ; Fri, 25 Sep 2020 17:15:24 +0200 (CEST) IronPort-SDR: GvIWvgRCRksifPkamrwUW6wPBid00pafPbhjCgHiZMMHWqhnW1L/79yCUhxe97yesrfFNBwdgH au1+cA8bYc3Q== X-IronPort-AV: E=McAfee;i="6000,8403,9755"; a="149323226" X-IronPort-AV: E=Sophos;i="5.77,302,1596524400"; d="scan'208";a="149323226" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2020 08:15:22 -0700 IronPort-SDR: O6kwT97/zYmEKtEseyganySMIR8fOr9c/W6rmNhvRCRhwDUFHCCdrK3yDlnUlDFtwtYIQbFReu WMKe9E3vj6cw== X-IronPort-AV: E=Sophos;i="5.77,302,1596524400"; d="scan'208";a="310847256" Received: from aburakov-mobl.ger.corp.intel.com (HELO [10.213.228.71]) ([10.213.228.71]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2020 08:15:21 -0700 To: =?UTF-8?B?6ZmI5q+F5by6?= , dev@dpdk.org References: <2009b3d5.1b5b.1730d6dd9cb.Coremail.chenyq3@wangsu.com> From: "Burakov, Anatoly" Message-ID: <2f373d6c-5b75-886f-007d-badd651b4a55@intel.com> Date: Fri, 25 Sep 2020 16:15:19 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <2009b3d5.1b5b.1730d6dd9cb.Coremail.chenyq3@wangsu.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] =?utf-8?q?Question_about_=E2=80=9CPCI_memorymmap?= =?utf-8?b?4oCd?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 02-Jul-20 3:49 AM, 陈毅强 wrote: > Hi,ALL > > We fail to start dpdk in a newly deployed machine.Failed progress works in proc-type=secondary mode. (DPDK version 16.04).Some logs are shown below > > ========================================================================================= > > Primary progress: > > EAL: PCI device 0000:04:00.1 on NUMA socket 0 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: PCI memory mapped at 0x7ffff5c04000 > EAL: PCI memory mapped at 0x7ffff6004000 > PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4 > PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb > EAL: PCI device 0000:82:00.0 on NUMA socket 1 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: PCI memory mapped at 0x7fffec3ff000 > EAL: PCI memory mapped at 0x7ffff7ff4000 > PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3 > PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb > EAL: PCI device 0000:82:00.1 on NUMA socket 1 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: PCI memory mapped at 0x7ffed25d9000 > EAL: PCI memory mapped at 0x7ffff7ff0000 > > > > > Secondary progress logs: > > > PMD: ixgbe_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0. > EAL: PCI device 0000:04:00.1 on NUMA socket 0 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: PCI memory mapped at 0x7ffff5c04000 > EAL: PCI memory mapped at 0x7ffff6004000 > PMD: ixgbe_set_tx_function(): Using simple tx code path > PMD: ixgbe_set_rx_function(): Port[1] doesn't meet Vector Rx preconditions or RTE_IXGBE_INC_VECTOR is not enabled > PMD: ixgbe_set_rx_function(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1. > EAL: PCI device 0000:82:00.0 on NUMA socket 1 > EAL: probe driver: 8086:10fb rte_ixgbe_pmd > EAL: PCI memory mapped at 0x7ffff65d8000 > EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:82:00.0/resource0 to address: 0x7fffec3ff000 > EAL: Error - exiting with code: 1 > Cause: Requested device 0000:82:00.0 cannot be used > > ================================================================================== > > Relevant code is list be below: > > > > > pci_uio_map_secondary(struct rte_pci_device *dev) > { > int fd, i; > struct mapped_pci_resource *uio_res; > struct mapped_pci_res_list *uio_res_list = > RTE_TAILQ_CAST(rte_uio_tailq.head, mapped_pci_res_list); > > > TAILQ_FOREACH(uio_res, uio_res_list, next) { > > > /* skip this element if it doesn't match our PCI address */ > if (rte_eal_compare_pci_addr(&uio_res->pci_addr, &dev->addr)) > continue; > > > for (i = 0; i != uio_res->nb_maps; i++) { > /* > * open devname, to mmap it > */ > fd = open(uio_res->maps[i].path, O_RDWR); > if (fd < 0) { > RTE_LOG(ERR, EAL, "Cannot open %s: %s\n", > uio_res->maps[i].path, strerror(errno)); > return -1; > } > > > void *mapaddr = pci_map_resource(uio_res->maps[i].addr, > fd, (off_t)uio_res->maps[i].offset, > (size_t)uio_res->maps[i].size, 0); > /* fd is not needed in slave process, close it */ > close(fd); > if (mapaddr != uio_res->maps[i].addr) { > RTE_LOG(ERR, EAL, > "Cannot mmap device resource file %s to address: %p\n", > uio_res->maps[i].path, > uio_res->maps[i].addr); > return -1; > } > } > return 0; > } > > > RTE_LOG(ERR, EAL, "Cannot find resource for device\n"); > return 1; > } > > > > > > > > And We check the system config . ASLR is not enabled on our system. I wondered why mapaddr != uio_res->maps[i].addr .Can anyone tell me why? > > > > > > > > Thanks in advance. > Hi, This is a known issue in DPDK and unfortunately it's not really solvable. I've made attempts in the past to address it with an RFC [1] patchset to enable deterministic hotplug, but it seems that this isn't a problem the majority care about enough to fix it. You might want to try with different --base-virtaddr values, but the root cause is that we do not control our memory address space, kernel does. Since secondary processes are predicated upon duplicating primary process's address space, this inevitably means that sometimes the duplication isn't possible because something else has already taken the address, and we cannot do anything about it. Enabling/disabling ASLR will not help or hurt with this issue; rather, it'll make it more deterministic (i.e. it'll either always work or always fail). So, playing around with various values for --base-virtaddr is pretty much your only option at this point. [1] http://patches.dpdk.org/bundle/aburakov/reliable_device_map/ -- Thanks, Anatoly