From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id EF652A058B; Wed, 25 Mar 2020 22:05:34 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E6A652C15; Wed, 25 Mar 2020 22:05:33 +0100 (CET) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by dpdk.org (Postfix) with ESMTP id ADD0D2BAE for ; Wed, 25 Mar 2020 22:05:31 +0100 (CET) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 02PL3hBZ099225; Wed, 25 Mar 2020 17:05:28 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 2ywd8exrpr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 25 Mar 2020 17:05:28 -0400 Received: from m0098396.ppops.net (m0098396.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 02PL3o9U100047; Wed, 25 Mar 2020 17:05:27 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 2ywd8exrnj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 25 Mar 2020 17:05:27 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.27/8.16.0.27) with SMTP id 02PL4AW4027173; Wed, 25 Mar 2020 21:05:26 GMT Received: from b01cxnp22036.gho.pok.ibm.com (b01cxnp22036.gho.pok.ibm.com [9.57.198.26]) by ppma05wdc.us.ibm.com with ESMTP id 2ywawfmc3u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 25 Mar 2020 21:05:26 +0000 Received: from b01ledav003.gho.pok.ibm.com (b01ledav003.gho.pok.ibm.com [9.57.199.108]) by b01cxnp22036.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 02PL5PuX13959884 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 25 Mar 2020 21:05:25 GMT Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AAEAEB205F; Wed, 25 Mar 2020 21:05:25 +0000 (GMT) Received: from b01ledav003.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CB521B2066; Wed, 25 Mar 2020 21:05:24 +0000 (GMT) Received: from ltc.linux.ibm.com (unknown [9.16.170.189]) by b01ledav003.gho.pok.ibm.com (Postfix) with ESMTP; Wed, 25 Mar 2020 21:05:24 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Wed, 25 Mar 2020 14:05:24 -0700 From: dwilder To: Jerin Jacob Cc: Aaron Conole , Michael Santana , Thomas Monjalon , Ferruh Yigit , Andrew Rybchenko , dpdk-dev , "Ruifeng Wang (Arm Technology China)" , David Marchand , David Christensen , David Wilder In-Reply-To: References: <20200220225207.30411-1-dwilder@us.ibm.com> <20200220225207.30411-2-dwilder@us.ibm.com> Message-ID: X-Sender: dwilder@us.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.645 definitions=2020-03-25_11:2020-03-24, 2020-03-25 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2003250159 Subject: Re: [dpdk-dev] [PATCH v3 1/3] eal/linux: select iova-mode va with no-huge option X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 2020-03-23 23:19, Jerin Jacob wrote: > On Mon, Mar 23, 2020 at 11:11 PM dwilder wrote: >> >> Thanks you for your review Jerin. See my responses are inline. >> >> On 2020-03-20 06:24, Jerin Jacob wrote: >> > On Fri, Feb 21, 2020 at 4:22 AM David Wilder >> > wrote: >> >> >> >> If --no-huge is set and iova-mode has not been specified force VA >> >> mode. >> >> If --no-huge and --iova-mode=PA is requested error out as this is >> >> an impossible configuration. >> >> >> >> Signed-off-by: David Wilder >> >> --- >> >> lib/librte_eal/linux/eal/eal.c | 14 ++++++++++++++ >> >> 1 file changed, 14 insertions(+) >> >> >> >> diff --git a/lib/librte_eal/linux/eal/eal.c >> >> b/lib/librte_eal/linux/eal/eal.c >> >> index 9530ee55f..d3a0a1731 100644 >> >> --- a/lib/librte_eal/linux/eal/eal.c >> >> +++ b/lib/librte_eal/linux/eal/eal.c >> >> @@ -1062,9 +1062,16 @@ rte_eal_init(int argc, char **argv) >> >> >> >> /* if no EAL option "--iova-mode=", use bus IOVA scheme >> >> */ >> >> if (internal_config.iova_mode == RTE_IOVA_DC) { >> >> + >> >> /* autodetect the IOVA mapping mode */ >> >> enum rte_iova_mode iova_mode = >> >> rte_bus_get_iommu_class(); >> >> >> >> + if (iova_mode == RTE_IOVA_PA && >> >> !rte_eal_has_hugepages()) { >> >> + iova_mode = RTE_IOVA_VA; >> >> > >> > What if igb_uio or vfio_nommu has been loaded(i.e no iommu support >> > enabled from the driver)? This would fail. >> >> Yes they would fail. If igb_uio or vfio_nommu (or any driver) cant be >> forced to VA mode it cant be used with out hugepages. Drivers can be >> available but not used therefor we print a warning message. > > I think, the warning will not be enough as the system will fail anyway. > > iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == > 1 > case, we need to return error. > > iova_mode == RTE_IOVA_PA && rte_eal_has_hugepages() == 0 && no_iommu == > 0 > case warning is enough. > I have a simpler solution. The goal here is to make --no-huge work when at least one bus/driver wants PA mode. A user can always override the selected mode with --iova-mode=va. So why not just make --no-huge the same as "--no-huge --iovs-mode=va" ? I am thinking: @@ -1060,6 +1060,11 @@ rte_eal_init(int argc, char **argv) phys_addrs = rte_eal_using_phys_addrs() != 0; + if (!phys_addrs) { + internal_config.iova_mode = RTE_IOVA_VA; + RTE_LOG(INFO, EAL, "Physical addresses are unavailable, selecting IOVA as VA mode.\n"); + } + /* if no EAL option "--iova-mode=", use bus IOVA scheme */ if (internal_config.iova_mode == RTE_IOVA_DC) { /* autodetect the IOVA mapping mode */ If a device cant run in VA mode, it will fail to initialize and report why. >> >> > >> >> + RTE_LOG(WARNING, EAL, "Some buses want 'PA' >> >> but forcing 'VA' because --no-huge is requested.\n"); >> >> + RTE_LOG(WARNING, EAL, "Not all buses may be >> >> able to initialize.\n"); >> >> + } >> >> + >> >> if (iova_mode == RTE_IOVA_DC) { >> >> RTE_LOG(DEBUG, EAL, "Buses did not request a >> >> specific IOVA mode.\n"); >> >> >> >> @@ -1111,6 +1118,13 @@ rte_eal_init(int argc, char **argv) >> >> internal_config.iova_mode; >> >> } >> >> >> >> + if (rte_eal_iova_mode() == RTE_IOVA_PA && >> >> + rte_eal_has_hugepages() == 0) { >> >> + rte_eal_init_alert("Cannot use IOVA as 'PA' with >> >> --no-huge"); >> > >> > Top of the tree already detecting this case. am I missing anything? >> > >> > [master]dell[dpdk.org] $ sudo ./build/app/test/dpdk-test -c 0x3 >> > --no-huge --iova-mode=pa >> > EAL: Detected 56 lcore(s) >> > EAL: Detected 2 NUMA nodes >> > EAL: Static memory layout is selected, amount of reserved memory can >> > be adjusted with -m or --socket-mem >> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket >> > EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not >> > available >> > EAL: Cannot use IOVA as 'PA' since physical addresses are not available >> > >> >> The check you reference is reporting that physical address are not >> available, for example no permissions to read /proc/self/pagemap. In >> this case, if --no-huge is set then PA mode is not allowed. There is >> no >> guarantee that physical address are persistent with out using >> hugepages. > > Since this check is under the following, Yes, make sense for the check. > The old command has explicit --iova-mode=pa. So it is in the > different code paths. > > /* if no EAL option "--iova-mode=", use bus IOVA scheme */ > if (internal_config.iova_mode == RTE_IOVA_DC) { > >> >> >> >> + rte_errno = EINVAL; >> >> + return -1; >> >> + } >> >> + >> >> if (rte_eal_iova_mode() == RTE_IOVA_PA && !phys_addrs) { >> >> rte_eal_init_alert("Cannot use IOVA as 'PA' since >> >> physical addresses are not available"); >> >> rte_errno = EINVAL; >> >> -- >> >> 2.25.0 >> >>