From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by dpdk.org (Postfix) with ESMTP id CF3292BAE for ; Thu, 20 Apr 2017 17:15:47 +0200 (CEST) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v3KFEN6f088984 for ; Thu, 20 Apr 2017 11:15:46 -0400 Received: from e18.ny.us.ibm.com (e18.ny.us.ibm.com [129.33.205.208]) by mx0a-001b2d01.pphosted.com with ESMTP id 29xgvg5v7x-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Thu, 20 Apr 2017 11:15:46 -0400 Received: from localhost by e18.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 20 Apr 2017 11:15:44 -0400 Received: from b01cxnp23033.gho.pok.ibm.com (9.57.198.28) by e18.ny.us.ibm.com (146.89.104.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 20 Apr 2017 11:15:42 -0400 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23033.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v3KFFjhZ39911442 for ; Thu, 20 Apr 2017 15:15:45 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9F39228099 for ; Thu, 20 Apr 2017 11:15:36 -0400 (EDT) Received: from d50lp02.ny.us.ibm.com (unknown [146.89.104.208]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTPS id 93429280A2 for ; Thu, 20 Apr 2017 11:15:36 -0400 (EDT) Received: from localhost by d50lp02.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 20 Apr 2017 11:15:41 -0400 Received: from smtp.notes.na.collabserv.com (192.155.248.82) by d50lp02.ny.us.ibm.com (158.87.18.21) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128/128) Thu, 20 Apr 2017 11:15:40 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Thu, 20 Apr 2017 15:15:39 -0000 Received: from us1a3-smtp07.a3.dal06.isc4sb.com (10.146.103.14) by smtp.notes.na.collabserv.com (10.106.227.105) with smtp.notes.na.collabserv.com ESMTP; Thu, 20 Apr 2017 15:15:36 -0000 Received: from us1a3-mail173.a3.dal06.isc4sb.com ([10.146.71.126]) by us1a3-smtp07.a3.dal06.isc4sb.com with ESMTP id 2017042015153560-383120 ; Thu, 20 Apr 2017 15:15:35 +0000 MIME-Version: 1.0 In-Reply-To: <6e669e2b-2cfd-078d-b6b0-5c3819fad796@ozlabs.ru> To: Alexey Kardashevskiy Cc: dev@dpdk.org, Adrian Schuepbach , Gowrishankar Muthukrishnan From: "Jonas Pfefferle1" Date: Thu, 20 Apr 2017 17:15:56 +0200 References: <20170420072402.38106-1-aik@ozlabs.ru> <20170420072402.38106-6-aik@ozlabs.ru> <12566b0a-8f9a-4040-a37d-2a106e49adcf@ozlabs.ru> <6e669e2b-2cfd-078d-b6b0-5c3819fad796@ozlabs.ru> X-KeepSent: 7556A259:43C475ED-C1258108:00537FA9; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1 October 14, 2013 X-LLNOutbound: False X-Disclaimed: 18743 X-TNEFEvaluated: 1 x-cbid: 17042015-0044-0000-0000-0000030E0FFD X-IBM-SpamModules-Scores: BY=0.292222; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.444167; ST=0; TS=0; UL=0; ISC=; MB=0.123996 X-IBM-SpamModules-Versions: BY=3.00006945; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00850128; UDB=6.00419851; IPR=6.00628758; BA=6.00005305; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015110; XFM=3.00000013; UTC=2017-04-20 15:15:37 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-04-20 14:51:27 - 6.00006625 x-cbparentid: 17042015-5102-0000-0000-000039A6ADF0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused X-TM-AS-GCONF: 00 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006945; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00850128; UDB=6.00419851; IPR=6.00628758; BA=6.00005305; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015110; XFM=3.00000013; UTC=2017-04-20 15:15:43 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-04-20_14:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=18 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704200122 X-Mailman-Approved-At: Fri, 21 Apr 2017 03:50:44 +0200 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus addresses for DMA map X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Apr 2017 15:15:48 -0000 Alexey Kardashevskiy wrote on 20/04/2017 16:22:01: > From: Alexey Kardashevskiy > To: Jonas Pfefferle1 > Cc: dev@dpdk.org, Gowrishankar Muthukrishnan > , Adrian Schuepbach > Date: 20/04/2017 16:22 > Subject: Re: [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus > addresses for DMA map > > On 20/04/17 23:25, Alexey Kardashevskiy wrote: > > On 20/04/17 19:04, Jonas Pfefferle1 wrote: > >> Alexey Kardashevskiy wrote on 20/04/2017 09:24:02: > >> > >>> From: Alexey Kardashevskiy > >>> To: dev@dpdk.org > >>> Cc: Alexey Kardashevskiy , JPF@zurich.ibm.com, > >>> Gowrishankar Muthukrishnan > >>> Date: 20/04/2017 09:24 > >>> Subject: [PATCH dpdk 5/5] RFC: vfio/ppc64/spapr: Use correct bus > >>> addresses for DMA map > >>> > >>> VFIO=5FIOMMU=5FSPAPR=5FTCE=5FCREATE ioctl() returns the actual bus ad= dress for > >>> just created DMA window. It happens to start from zero because the default > >>> window is removed (leaving no windows) and new window starts from zero. > >>> However this is not guaranteed and the new window may start from another > >>> address, this adds an error check. > >>> > >>> Another issue is that IOVA passed to VFIO=5FIOMMU=5FMAP=5FDMA should = be a PCI > >>> bus address while in this case a physical address of a user page is used. > >>> This changes IOVA to start from zero in a hope that the rest of DPDK > >>> expects this. > >> > >> This is not the case. DPDK expects a 1:1 mapping PA=3D=3DIOVA. It will= use the > >> phys=5Faddr of the memory segment it got from /proc/self/pagemap cf. > >> librte=5Feal/linuxapp/eal/eal=5Fmemory.c. We could try setting it here= to the > >> actual iova which basically makes the whole virtual to phyiscal mapping > >> with pagemap unnecessary which I believe should be the case for VFIO > >> anyway. Pagemap should only be needed when using pci=5Fuio. > > > > > > Ah, ok, makes sense now. But it sure needs a big fat comment there as it is > > not obvious why host RAM address is used there as DMA window start is not > > guaranteed. > > Well, either way there is some bug - ms[i].phys=5Faddr and ms[i].addr=5F64 both > have exact same value, in my setup it is 3fffb33c0000 which is a userspace > address - at least ms[i].phys=5Faddr must be physical address. This might be the case if you are not using hugetlbfs i.e. passing "--no-huge" cf. eal=5Fmemory.c:980 /* hugetlbfs can be disabled */ if (internal=5Fconfig.no=5Fhugetlbfs) { addr =3D mmap(NULL, internal=5Fconfig.memory, PROT=5FREAD | PROT=5FWRITE, MAP=5FPRIVATE | MAP=5FANONYMOUS, 0, 0); if (addr =3D=3D MAP=5FFAILED) { RTE=5FLOG(ERR, EAL, "%s: mmap() failed: %s\n", =5F=5Ffunc=5F=5F, strerror(errno)); return -1; } mcfg->memseg[0].phys=5Faddr =3D (phys=5Faddr=5Ft)(uintptr=5Ft)addr; mcfg->memseg[0].addr =3D addr; mcfg->memseg[0].hugepage=5Fsz =3D RTE=5FPGSIZE=5F4K; mcfg->memseg[0].len =3D internal=5Fconfig.memory; mcfg->memseg[0].socket=5Fid =3D 0; return 0; } If it fails to get the virt2phys mapping it actually assigns iovas starting from 0 to the memory segments, cf. set=5Fphysaddrs eal=5Fmemory.c:263 > > > > > > > >> > >>> > >>> Signed-off-by: Alexey Kardashevskiy > >>> --- > >>> lib/librte=5Feal/linuxapp/eal/eal=5Fvfio.c | 12 ++++++++++-- > >>> 1 file changed, 10 insertions(+), 2 deletions(-) > >>> > >>> diff --git a/lib/librte=5Feal/linuxapp/eal/eal=5Fvfio.c b/lib/ > >>> librte=5Feal/linuxapp/eal/eal=5Fvfio.c > >>> index 46f951f4d..8b8e75c4f 100644 > >>> --- a/lib/librte=5Feal/linuxapp/eal/eal=5Fvfio.c > >>> +++ b/lib/librte=5Feal/linuxapp/eal/eal=5Fvfio.c > >>> @@ -658,7 +658,7 @@ vfio=5Fspapr=5Fdma=5Fmap(int vfio=5Fcontainer=5Ff= d) > >>> { > >>> const struct rte=5Fmemseg *ms =3D rte=5Feal=5Fget=5Fphysmem=5Flay= out(); > >>> int i, ret; > >>> - > >>> + phys=5Faddr=5Ft io=5Foffset; > >>> struct vfio=5Fiommu=5Fspapr=5Fregister=5Fmemory reg =3D { > >>> .argsz =3D sizeof(reg), > >>> .flags =3D 0 > >>> @@ -702,6 +702,13 @@ vfio=5Fspapr=5Fdma=5Fmap(int vfio=5Fcontainer=5F= fd) > >>> return -1; > >>> } > >>> > >>> + io=5Foffset =3D create.start=5Faddr; > >>> + if (io=5Foffset) { > >>> + RTE=5FLOG(ERR, EAL, " DMA offsets other than zero is not > supported, " > >>> + "new window is created at %lx\n", io=5Foffset); > >>> + return -1; > >>> + } > >>> + > >>> /* map all DPDK segments for DMA. use 1:1 PA to IOVA mapping */ > >>> for (i =3D 0; i < RTE=5FMAX=5FMEMSEG; i++) { > >>> struct vfio=5Fiommu=5Ftype1=5Fdma=5Fmap dma=5Fmap; > >>> @@ -723,7 +730,7 @@ vfio=5Fspapr=5Fdma=5Fmap(int vfio=5Fcontainer=5Ff= d) > >>> dma=5Fmap.argsz =3D sizeof(struct vfio=5Fiommu=5Ftype1=5Fdma= =5Fmap); > >>> dma=5Fmap.vaddr =3D ms[i].addr=5F64; > >>> dma=5Fmap.size =3D ms[i].len; > >>> - dma=5Fmap.iova =3D ms[i].phys=5Faddr; > >>> + dma=5Fmap.iova =3D io=5Foffset; > >>> dma=5Fmap.flags =3D VFIO=5FDMA=5FMAP=5FFLAG=5FREAD | > >>> VFIO=5FDMA=5FMAP=5FFLAG=5FWRITE; > >>> > >>> @@ -735,6 +742,7 @@ vfio=5Fspapr=5Fdma=5Fmap(int vfio=5Fcontainer=5Ff= d) > >>> return -1; > >>> } > >>> > >>> + io=5Foffset +=3D dma=5Fmap.size; > >>> } > >>> > >>> return 0; > >>> -- > >>> 2.11.0 > >>> > >> > > > > > > > -- > Alexey >