From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id E0715A10DA for ; Thu, 1 Aug 2019 06:12:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id D4D6F1C011; Thu, 1 Aug 2019 06:12:47 +0200 (CEST) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 915391BFE0 for ; Thu, 1 Aug 2019 06:12:45 +0200 (CEST) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x714CNU3029692 for ; Thu, 1 Aug 2019 00:12:45 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [158.85.210.108]) by mx0b-001b2d01.pphosted.com with ESMTP id 2u3pa8d6sr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 01 Aug 2019 00:12:44 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Thu, 1 Aug 2019 04:12:43 -0000 Received: from us1b3-smtp03.a3dr.sjc01.isc4sb.com (10.122.7.173) by smtp.notes.na.collabserv.com (10.122.47.46) with smtp.notes.na.collabserv.com ESMTP; Thu, 1 Aug 2019 04:12:40 -0000 Received: from us1b3-mail157.a3dr.sjc03.isc4sb.com ([10.160.174.125]) by us1b3-smtp03.a3dr.sjc01.isc4sb.com with ESMTP id 2019080104124044-27696 ; Thu, 1 Aug 2019 04:12:40 +0000 In-Reply-To: From: "Takeshi T Yoshimura" To: David Marchand Cc: dev , David Christensen Date: Thu, 1 Aug 2019 04:12:40 +0000 MIME-Version: 1.0 Sensitivity: Importance: Normal X-Priority: 3 (Normal) References: , <20190731033523.2482-1-tyos@jp.ibm.com> X-Mailer: IBM iNotes ($HaikuForm 1054) | IBM Domino Build SCN1812108_20180501T0841_FP55 May 22, 2019 at 11:09 X-KeepSent: 106EBA51:3EB21A96-00258449:001721EF; type=4; name=$KeepSent X-LLNOutbound: False X-Disclaimed: 59671 X-TNEFEvaluated: 1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 x-cbid: 19080104-3017-0000-0000-0000007B453F X-IBM-SpamModules-Scores: BY=0.283104; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.415104; ST=0; TS=0; UL=0; ISC=; MB=0.052731 X-IBM-SpamModules-Versions: BY=3.00011532; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000287; SDB=6.01240366; UDB=6.00654073; IPR=6.01021790; MB=3.00027990; MTD=3.00000008; XFM=3.00000015; UTC=2019-08-01 04:12:43 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2019-07-31 23:23:20 - 6.00010233 x-cbparentid: 19080104-3018-0000-0000-000000C88D10 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-08-01_02:, , signatures=0 X-Proofpoint-Spam-Reason: safe Subject: Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in ppc X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" -----David Marchand wrote: ----- >To: Takeshi T Yoshimura >From: David Marchand >Date: 08/01/2019 04:45AM >Cc: dev , David Christensen >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA in >ppc > >On Wed, Jul 31, 2019 at 12:44 PM Takeshi T Yoshimura > wrote: >> >> -----David Marchand wrote: ----- >> >> >To: Takeshi Yoshimura >> >From: David Marchand >> >Date: 07/31/2019 06:29PM >> >Cc: dev >> >Subject: [EXTERNAL] Re: [dpdk-dev] [PATCH] eal: forcing IOVA as PA >in >> >ppc >> > >> >On Wed, Jul 31, 2019 at 5:36 AM Takeshi Yoshimura > >> >wrote: >> >> >> >> Commit b76fafb174d2 ("eal: fix IOVA mode selection as VA for PCI >> >> drivers") breaks ppc apps with no IOVA configs (or RTE=5FIOVA=5FDC) >> >> because of the inconsistency of user's request and the result of >> >> device capability for IOVA mode. I updated the code to force >IOVA >> >as >> >> PA in ppc as before because current ppc driver does not support >VA >> >> mode. >> > >> >What is the "ppc driver" you are referring to? >> >Do you mean the vfio implementation for ppc? >> > >> >> >> >> Theoretically, ppc can support VA mode, but I suspect that ppc >with >> >> VA mode may have performance issues to create a big DMA window >> >> (VA often uses higher addresses than PA). So, I didn't change >the >> >> code to check device capability in ppc. >> > >> >Sorry, I am confused. >> >Is VA mode not working at all? or is it a performance issue? >> > >> > >> >-- >> >David Marchand >> > >> > >> >> Ah, I overlooked your first question. The "ppc driver" is in >> drivers/bus/pci/linux/pci.c. In pci.c, >pci=5Fone=5Fdevice=5Fiommu=5Fsupport=5Fva() >> always returns false in ppc. so, the driver does not allow VA mode >in ppc. >> >> This return value is passed to rte=5Fpci=5Fprobe=5Fone=5Fdriver() >> in drivers/bus/pci/pci=5Fcommon.c. And the error log appeared like: >> >> EAL: PCI device 0000:01:00.0 on NUMA socket 0 >> EAL: probe driver: 144d:a822 spdk=5Fnvme >> EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not >initializing > >Can you confirm on which kernel driver this device was bound ? >I'd like to be sure this is the same issue than David. > > >--=20 >David Marchand > > Here's the full debug log I got. I ran upstream SPDK on NVMe on IBM Power9 = AC922. The operating system is Ubuntu 18.04 and Linux 4.17. Starting SPDK v19.07-pre / DPDK 19.08.0-rc3 initialization... [ DPDK EAL parameters: identify --no-shconf -c 0x1 -n 1 -m 0 --log-level=3D= lib.eal:6 --log-level=3Dlib.cryptodev:5 --log-level=3Duser1:6 --log-level= =3Dlib.eal:8 --base-virtaddr=3D0x200000000000 --match-allocations --file-pr= efix=3Dspdk=5Fpid139059 ] EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 7 (sPAPR) is supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x30000 bytes EAL: Virtual area found at 0x200000000000 (size =3D 0x30000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket=5Fid:0 hugepage=5Fsz:2097152 EAL: Detected memory type: socket=5Fid:8 hugepage=5Fsz:2097152 EAL: Creating 4 segment lists: n=5Fsegs:8192 socket=5Fid:0 hugepage=5Fsz:20= 97152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200000030000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200000200000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200400200000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200400400000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200800400000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200800600000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x200c00600000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 0 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x200c00800000 (size =3D 0x400000000) EAL: Creating 4 segment lists: n=5Fsegs:8192 socket=5Fid:8 hugepage=5Fsz:20= 97152 EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201000800000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201000a00000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201400a00000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201400c00000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201800c00000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201800e00000 (size =3D 0x400000000) EAL: Ask a virtual area of 0x70000 bytes EAL: Virtual area found at 0x201c00e00000 (size =3D 0x70000) EAL: Memseg list allocated: 0x800kB at socket 8 EAL: Ask a virtual area of 0x400000000 bytes EAL: Virtual area found at 0x201c01000000 (size =3D 0x400000000) EAL: Hugepages will be freed exactly as allocated. EAL: No shared files mode enabled, IPC is disabled EAL: TSC frequency is ~510000 KHz EAL: Master lcore 0 is ready (tid=3D77e7345a44a0;cpuset=3D[0]) EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL=5FPREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: request: mp=5Fmalloc=5Fsync EAL: No shared files mode enabled, IPC is disabled EAL: Heap on socket 0 was expanded by 2MB EAL: No shared files mode enabled, IPC is disabled EAL: Mem event callback 'spdk:(nil)' registered EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk=5Fnvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0000:01:00.0 cannot be used EAL: PCI device 0003:01:00.0 on NUMA socket 0 EAL: probe driver: 144d:a822 spdk=5Fnvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0003:01:00.0 cannot be used EAL: PCI device 0030:01:00.0 on NUMA socket 8 EAL: probe driver: 144d:a822 spdk=5Fnvme EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing EAL: Requested device 0030:01:00.0 cannot be used