From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out1134-178.mail.aliyun.com (out1134-178.mail.aliyun.com [42.120.134.178]) by dpdk.org (Postfix) with ESMTP id 2A4842A9 for ; Fri, 7 Nov 2014 15:28:31 +0100 (CET) X-Alimail-AntiSpam: AC=CONTINUE; BC=0.02341166|-1; FP=0|0|0|0|0|-1|-1|-1; HT=r46d02010; MF=liang.xu@cinfotech.cn; PH=DW; RN=4; RT=4; SR=0; Received: from WS-web (liang.xu@cinfotech.cn[222.65.239.251]) by r41g08149.xy2.aliyun.com at Fri, 07 Nov 2014 22:37:47 +0800 Date: Fri, 07 Nov 2014 22:37:47 +0800 From: "XU Liang" To: "=?UTF-8?B?QnVyYWtvdiwgQW5hdG9seQ==?=" , "dev@dpdk.org" Message-ID: <810e3873-2f78-4c24-9255-59c7ba401390@cinfotech.cn> X-Mailer: Alimail-Mailagent revision 2667797 MIME-Version: 1.0 References: <1415193919-17361-1-git-send-email-liang.xu@cinfotech.cn> <1415347284-15468-1-git-send-email-liang.xu@cinfotech.cn>, C6ECDF3AB251BE4894318F4E4512369780C07822@IRSMSX109.ger.corp.intel.com <2c2ad9a4-1c04-4641-9667-249617ae2c56@cinfotech.cn>, C6ECDF3AB251BE4894318F4E4512369780C07848@IRSMSX109.ger.corp.intel.com In-Reply-To: C6ECDF3AB251BE4894318F4E4512369780C07848@IRSMSX109.ger.corp.intel.com x-aliyun-mail-creator: Webmail4_2670074_M3LTW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTBfMTBfMSkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzM4LjAuMjEyNS4xMTEgU2FmYXJpLzUzNy4zNg==vN Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] =?utf-8?q?=5BPATCH_v6=5D_eal=3A_map_uio_resources_afte?= =?utf-8?q?r_hugepages_when_the_base=5Fvirtaddr_is_configured=2E?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list Reply-To: XU Liang List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 14:28:38 -0000 Because vfio and uio can be used at the same time, we need save the value of r= equested_addr at=C2=A0pci_vfio_map_resource() and=C2=A0pci_uio_map_resource().= Your guys like add a=C2=A0global=C2=A0variable "void * pci_requested_addr" or = a "void** requested_addr" parameter at=C2=A0pci_vfio_map_resource() and=C2=A0p= ci_uio_map_resource() ?-------------------------------------------------------= -----------From:Burakov, Anatoly Time:2014 Nov 7 (F= ri) 18 : 03To:=E5=BE=90=E4=BA=AE , dev@dpdk.org Cc:thomas.monjalon@6wind.com , De Lara Guarc= h, Pablo Subject:RE: [PATCH v6] eal: map uio r= esources after hugepages when the base_virtaddr is configured.=0A=0A=0A=0Afont= -family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS Gothic;panose= -1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2= 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;pan= ose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 = 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0in;margin-bottom: 1= .0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}=0Aa:link, spa= n.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underli= ne;}=0Aa:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #95= 4f72;text-decoration: underline;}=0Aspan.EmailStyle17 {mso-style-type: persona= l-reply;font-family: Calibri , sans-serif;color: #1f497d;}=0A*.MsoChpDefault {= mso-style-type: export-only;font-family: Calibri , sans-serif;}=0Asize: 8.5in = 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page: WordSection1;}=0A= =0A=0A=0A=0AAh, yes, sorry about that, haven=E2=80=99t quite woke up yet=0AL Y= ou=E2=80=99re right. So it=E2=80=99s removing the dependency on --base-virtadd= r, moving the function to eal_pci.c=0A =C2=A0and adding similar VFIO code that= =E2=80=99s left then.=0A=C2=A0=0AThanks,=0AAnatoly=0A=C2=A0=0AFrom: XU Liang [= mailto:liang.xu@cinfotech.cn]=0A=0A=0ASent: Friday, November 7, 2014 9:57 AM=0A= =0ATo: Burakov, Anatoly; dev@dpdk.org=0A=0ACc: thomas.monjalon@6wind.com; De L= ara Guarch, Pablo=0A=0ASubject: Re: [PATCH v6] eal: map uio resources after hu= gepages when the base_virtaddr is configured.=0A=C2=A0=0A=0A=0AHow to find the= =C2=A0maximum end virtual address ?=C2=A0I'm not the DPDK expert, but I will t= ry to do my best.=0A=0A=0A=C2=A0=0A=0A=0AIf the segments isn't overlap, "if(se= g->addr > last->addr)=C2=A0last =3D seg;" already find the segment with=C2=A0m= aximum end virtual address.=0A=0A=0A=C2=A0=0A=0A=0A---------------------------= ---------------------------------------=0A=0A=0A=0AFrom:Burakov, Anatoly =0A=0A=0ATime:2014 Nov 7 (Fri) 17 : 47=0A=0A=0ATo:=E5=BE= =90=E4=BA=AE=0A , =0Adev@dpdk.org =0A=0A=0A= Cc:thomas.monjalon@6wind.com , De Lara Guarch, Pabl= o =0A=0A=0ASubject:RE: [PATCH v6] eal: map uio= resources after hugepages when the base_virtaddr is configured.=0A=0A=0A=C2=A0= =0A=0AThe commit message looks fine to me, but VFIO code needs to be adjusted = the same way.=0A=0A=0A=0AAlso, now that I think of it, you can't simply assume= that whatever last memseg you have has the latest virtual address. When IVSHM= EM is initialized, it too reserves some space in the virtual memory, which can= be higher than the last hugepage, but not be the=0A last hugepage (because IV= SHMEM memory is first to be reserved, before the main memory).=0A=0A=0A=0AMy a= dvice would be to rewrite the function to return the maximum end virtual addre= ss (instead of a last segment) and move it to eal_pci.c (and include declarati= on for it in include/eal_pci_init.h).=0A=0A=0A=0AMy apologies for not thinking= about any of this during the first 6 iterations of the patch :(=0A=0A=0A=0ATh= anks,=0A=0AAnatoly=0A=0A=0A=0A-----Original Message-----=0A=0AFrom: lxu [mailt= o:liang.xu@cinfotech.cn]=0A=0A=0ASent: Friday, November 7, 2014 8:01 AM=0A=0AT= o: dev@dpdk.org=0A=0ACc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara = Guarch, Pablo=0A=0ASubject: [PATCH v6] eal: map uio resources after hugepages = when the base_virtaddr is configured.=0A=0A=0A=0AA multiple process DPDK appli= cation must mmap hugepages and pci resource into same virtual addresses. By de= fault the virtual addresses chosen by the primary process automatically when c= alling the mmap. But sometime the virtual addresses chosen by the primary=0A p= rocess isn't usable at secondary process. Such as the secondary process linked= with more libraries than primary process. The library has been mapped into th= is virtual address. The command line parameter 'base-virtaddr' has been added = for this situation. If=0A it's configured, the hugepages will be mapped into t= his base address. But the virtual address of uio resource mapped still does no= t refer to the parameter. In that case it would still fail.=0A=0A=0A=0AThis pa= tch try to map uio resources after hugepages when the base_virtaddr is configu= red. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by= set base-virtaddr into free virtual address space.=0A=0A=0A=0ASigned-off-by: = lxu =0A=0A---=0A=0Alib/librte_eal/linuxapp/eal/eal_pci_= uio.c | 29 ++++++++++++++++++++++++++++-=0A=0A1 file changed, 28 insertions(+)= , 1 deletion(-)=0A=0A=0A=0Adiff --git a/lib/librte_eal/linuxapp/eal/eal_pci_ui= o.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c=0A=0Aindex 7e62266..a2c9ab6 10= 0644=0A=0A--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c=0A=0A+++ b/lib/librt= e_eal/linuxapp/eal/eal_pci_uio.c=0A=0A@@ -273,6 +273,24 @@ pci_get_uio_dev(str= uct rte_pci_device *dev, char *dstbuf,=0A=0Areturn uio_num;=0A=0A}=0A=0A=0A=0A= +static inline const struct rte_memseg *=0A=0A+get_physmem_last(void)=0A=0A+{=0A= =0A+ const struct rte_memseg * seg =3D rte_eal_get_physmem_layout();=0A=0A+ co= nst struct rte_memseg * last =3D seg;=0A=0A+ unsigned i =3D 0;=0A=0A+=0A=0A+ f= or (i=3D0; iaddr =3D=3D NULL)=0A= =0A+ break;=0A=0A+=0A=0A+ if(seg->addr > last->addr)=0A=0A+ last =3D seg;=0A=0A= +=0A=0A+ }=0A=0A+ return last;=0A=0A+}=0A=0A+=0A=0A/* map the PCI resource of = a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_devi= ce *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)= =0A=0Astruct mapped_pci_resource *uio_res;=0A=0Astruct pci_map *maps;=0A=0A=0A= =0A+ /* map uio resource into user required virtual address */=0A=0A+ static v= oid * requested_addr;=0A=0A+ if (internal_config.base_virtaddr && NULL =3D=3D = requested_addr) {=0A=0A+ const struct rte_memseg * last =3D get_physmem_last()= ;=0A=0A+ requested_addr =3D RTE_PTR_ADD(last->addr, last->len);=0A=0A+ }=0A=0A= +=0A=0Adev->intr_handle.fd =3D -1;=0A=0Adev->intr_handle.type =3D RTE_INTR_HAN= DLE_UNKNOWN;=0A=0A=0A=0A@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_= pci_device *dev)=0A=0Aif (maps[j].addr !=3D NULL)=0A=0Afail =3D 1;=0A=0Aelse {= =0A=0A- mapaddr =3D pci_map_resource(NULL, fd, (off_t)offset,=0A=0A+ mapaddr =3D= pci_map_resource(requested_addr, fd, (off_t)offset,=0A=0A(size_t)maps[j].size= );=0A=0Aif (mapaddr =3D=3D NULL)=0A=0Afail =3D 1;=0A=0A+ else if (NULL !=3D re= quested_addr)=0A=0A+ requested_addr =3D RTE_PTR_ADD(mapaddr, maps[j].size);=0A= =0A}=0A=0A=0A=0Aif (fail) {=0A=0A--=0A=0A1.9.1=0A=0A=0A >From bruce.richardson@intel.com Fri Nov 7 15:34:52 2014 Return-Path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id D90B92A9 for ; Fri, 7 Nov 2014 15:34:51 +0100 (CET) Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga101.jf.intel.com with ESMTP; 07 Nov 2014 06:44:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.07,332,1413270000"; d="scan'208";a="604153305" Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.32]) by orsmga001.jf.intel.com with SMTP; 07 Nov 2014 06:44:11 -0800 Received: by (sSMTP sendmail emulation); Fri, 07 Nov 2014 14:44:10 +0025 Date: Fri, 7 Nov 2014 14:44:10 +0000 From: Bruce Richardson To: jigsaw Message-ID: <20141107144410.GC12092@bricha3-MOBL3> References: <20141106092228.GA3056@bricha3-MOBL3> <9190772.1rnKUO3oNV@xps13> <545b6b74.a96db40a.26af.ffffe7fb@mx.google.com> <20141106135951.GB7252@bricha3-MOBL3> <20141107094521.GB4628@bricha3-MOBL3> <20141107135303.GB12092@bricha3-MOBL3> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Organization: Intel Shannon Ltd. User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] =?utf-8?b?562U5aSNOiAgW1BBVENIXSBBZGQgdXNlciBkZWZpbmVk?= =?utf-8?q?_tag_calculation_callback_tolibrte=5Fdistributor=2E?= X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Nov 2014 14:34:52 -0000 On Fri, Nov 07, 2014 at 04:31:18PM +0200, jigsaw wrote: > Hi Bruce, > > Pls have a quick look at the diff to see if this is exactly what you mean > about the bitmask. > I just wrote it without even compiling, just to express the idea. So it may > leave some places unpatched. > If this is agreed, I will make a decent test to verify it before sending > the patch for RFC. > > diff --git a/lib/librte_distributor/rte_distributor.c > b/lib/librte_distributor/rte_di > index 585ff88..d606bcf 100644 > --- a/lib/librte_distributor/rte_distributor.c > +++ b/lib/librte_distributor/rte_distributor.c > @@ -92,6 +92,8 @@ struct rte_distributor { > unsigned num_workers; /**< Number of workers > polling */ > > uint32_t in_flight_tags[RTE_MAX_LCORE]; > + uint32_t in_flight_bitmask; > + > struct rte_distributor_backlog backlog[RTE_MAX_LCORE]; > > union rte_distributor_buffer bufs[RTE_MAX_LCORE]; > @@ -188,6 +190,7 @@ static inline void > handle_worker_shutdown(struct rte_distributor *d, unsigned wkr) > { > d->in_flight_tags[wkr] = 0; > + d->in_flight_mask &= ~(1 << wkr); > d->bufs[wkr].bufptr64 = 0; > if (unlikely(d->backlog[wkr].count != 0)) { > /* On return of a packet, we need to move the > @@ -241,6 +244,7 @@ process_returns(struct rte_distributor *d) > else { > d->bufs[wkr].bufptr64 = RTE_DISTRIB_GET_BUF; > d->in_flight_tags[wkr] = 0; > + d->in_flight_mask &= ~(1 << wkr); > } > oldbuf = data >> RTE_DISTRIB_FLAG_BITS; > } else if (data & RTE_DISTRIB_RETURN_BUF) { > @@ -282,12 +286,13 @@ rte_distributor_process(struct rte_distributor *d, > next_mb = mbufs[next_idx++]; > next_value = (((int64_t)(uintptr_t)next_mb) > << RTE_DISTRIB_FLAG_BITS); > - new_tag = (next_mb->hash.rss | 1); > + new_tag = next_mb->hash.rss; > > uint32_t match = 0; > unsigned i; > for (i = 0; i < d->num_workers; i++) > - match |= (!(d->in_flight_tags[i] ^ new_tag) > + match |= (((!(d->in_flight_tags[i] ^ > new_tag)) & > + (d->in_flight_bitmask >> i)) I would not do the bitmask comparison here, as that's extra instruction in the loop. Instead, because its a bitmask, build up the match variable as it was before, and then just do a single and operation afterwards, outside the loop body. /Bruce > << i); > > if (match) { > @@ -309,6 +314,7 @@ rte_distributor_process(struct rte_distributor *d, > else { > d->bufs[wkr].bufptr64 = next_value; > d->in_flight_tags[wkr] = new_tag; > + d->in_flight_bitmask |= 1 << wkr; > next_mb = NULL; > } > oldbuf = data >> RTE_DISTRIB_FLAG_BITS; > > >