Re: [dpdk-dev] [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.

["XU Liang" <liang.xu@cinfotech.cn>]

Because vfio and uio can be used at the same time, we need save the value of requested_addr at pci_vfio_map_resource() and pci_uio_map_resource().Your guys like add a global variable "void * pci_requested_addr" or a "void** requested_addr" parameter at pci_vfio_map_resource() and pci_uio_map_resource() ?------------------------------------------------------------------From:Burakov, Anatoly <anatoly.burakov@intel.com>Time:2014 Nov 7 (Fri) 18 : 03To:徐亮 <liang.xu@cinfotech.cn>, dev@dpdk.org <dev@dpdk.org>Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



font-family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5 3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family: Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin: 0.0in;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman , serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color: #0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color: #954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri , sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 8.5in 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page: WordSection1;}




Ah, yes, sorry about that, haven’t quite woke up yet
L You’re right. So it’s removing the dependency on --base-virtaddr, moving the function to eal_pci.c
  and adding similar VFIO code that’s left then.
 
Thanks,
Anatoly
 
From: XU Liang [mailto:liang.xu@cinfotech.cn]


Sent: Friday, November 7, 2014 9:57 AM

To: Burakov, Anatoly; dev@dpdk.org

Cc: thomas.monjalon@6wind.com; De Lara Guarch, Pablo

Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.
 


How to find the maximum end virtual address ? I'm not the DPDK expert, but I will try to do my best.


 


If the segments isn't overlap, "if(seg->addr > last->addr) last = seg;" already find the segment with maximum end virtual address.


 


------------------------------------------------------------------



From:Burakov, Anatoly <anatoly.burakov@intel.com>


Time:2014 Nov 7 (Fri) 17 : 47


To:徐亮
 <liang.xu@cinfotech.cn>, 
dev@dpdk.org <dev@dpdk.org>


Cc:thomas.monjalon@6wind.com <thomas.monjalon@6wind.com>, De Lara Guarch, Pablo <pablo.de.lara.guarch@intel.com>


Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.


 

The commit message looks fine to me, but VFIO code needs to be adjusted the same way.



Also, now that I think of it, you can't simply assume that whatever last memseg you have has the latest virtual address. When IVSHMEM is initialized, it too reserves some space in the virtual memory, which can be higher than the last hugepage, but not be the
 last hugepage (because IVSHMEM memory is first to be reserved, before the main memory).



My advice would be to rewrite the function to return the maximum end virtual address (instead of a last segment) and move it to eal_pci.c (and include declaration for it in include/eal_pci_init.h).



My apologies for not thinking about any of this during the first 6 iterations of the patch :(



Thanks,

Anatoly



-----Original Message-----

From: lxu [mailto:liang.xu@cinfotech.cn]


Sent: Friday, November 7, 2014 8:01 AM

To: dev@dpdk.org

Cc: Burakov, Anatoly; thomas.monjalon@6wind.com; De Lara Guarch, Pablo

Subject: [PATCH v6] eal: map uio resources after hugepages when the base_virtaddr is configured.



A multiple process DPDK application must mmap hugepages and pci resource into same virtual addresses. By default the virtual addresses chosen by the primary process automatically when calling the mmap. But sometime the virtual addresses chosen by the primary
 process isn't usable at secondary process. Such as the secondary process linked with more libraries than primary process. The library has been mapped into this virtual address. The command line parameter 'base-virtaddr' has been added for this situation. If
 it's configured, the hugepages will be mapped into this base address. But the virtual address of uio resource mapped still does not refer to the parameter. In that case it would still fail.



This patch try to map uio resources after hugepages when the base_virtaddr is configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be resolved by set base-virtaddr into free virtual address space.



Signed-off-by: lxu <liang.xu@cinfotech.cn>

---

lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-

1 file changed, 28 insertions(+), 1 deletion(-)



diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

index 7e62266..a2c9ab6 100644

--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c

@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,

return uio_num;

}



+static inline const struct rte_memseg *

+get_physmem_last(void)

+{

+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();

+ const struct rte_memseg * last = seg;

+ unsigned i = 0;

+

+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {

+ if (seg->addr == NULL)

+ break;

+

+ if(seg->addr > last->addr)

+ last = seg;

+

+ }

+ return last;

+}

+

/* map the PCI resource of a PCI device in virtual memory */ int pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@ pci_uio_map_resource(struct rte_pci_device *dev)

struct mapped_pci_resource *uio_res;

struct pci_map *maps;



+ /* map uio resource into user required virtual address */

+ static void * requested_addr;

+ if (internal_config.base_virtaddr && NULL == requested_addr) {

+ const struct rte_memseg * last = get_physmem_last();

+ requested_addr = RTE_PTR_ADD(last->addr, last->len);

+ }

+

dev->intr_handle.fd = -1;

dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;



@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)

if (maps[j].addr != NULL)

fail = 1;

else {

- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,

+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,

(size_t)maps[j].size);

if (mapaddr == NULL)

fail = 1;

+ else if (NULL != requested_addr)

+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);

}



if (fail) {

--

1.9.1



From bruce.richardson@intel.com  Fri Nov  7 15:34:52 2014
Return-Path: <bruce.richardson@intel.com>
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
 by dpdk.org (Postfix) with ESMTP id D90B92A9
 for <dev@dpdk.org>; Fri,  7 Nov 2014 15:34:51 +0100 (CET)
Received: from orsmga001.jf.intel.com ([10.7.209.18])
 by orsmga101.jf.intel.com with ESMTP; 07 Nov 2014 06:44:14 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.07,332,1413270000"; d="scan'208";a="604153305"
Received: from bricha3-mobl3.ger.corp.intel.com ([10.243.20.32])
 by orsmga001.jf.intel.com with SMTP; 07 Nov 2014 06:44:11 -0800
Received: by  (sSMTP sendmail emulation); Fri, 07 Nov 2014 14:44:10 +0025
Date: Fri, 7 Nov 2014 14:44:10 +0000
From: Bruce Richardson <bruce.richardson@intel.com>
To: jigsaw <jigsaw@gmail.com>
Message-ID: <20141107144410.GC12092@bricha3-MOBL3>
References: <20141106092228.GA3056@bricha3-MOBL3> <9190772.1rnKUO3oNV@xps13>
 <545b6b74.a96db40a.26af.ffffe7fb@mx.google.com>
 <20141106135951.GB7252@bricha3-MOBL3>
 <CAHVfvh4U4PZKZSue_kKDQKATC2snb_\x10OD08LGmUtieBc_LzA@mail.gmail.com>
 <CAHVfvh5SzJ-kpQQ9h=1wmMihiitcJXeR9mcNa1in8x6Gb6tSqQ@mail.gmail.com>
 <20141107094521.GB4628@bricha3-MOBL3>
 <CAHVfvh6y4f7+bMhzmwOu5c0Y4wzwNaxj4sQPtq8cabGbdHrzXg@mail.gmail.com>
 <20141107135303.GB12092@bricha3-MOBL3>
 <CAHVfvh7ggGB_q1Rs1c3-9PRwDr_GKA+etaMXRSeKCfUKoUx8hQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAHVfvh7ggGB_q1Rs1c3-9PRwDr_GKA+etaMXRSeKCfUKoUx8hQ@mail.gmail.com>
Organization: Intel Shannon Ltd.
User-Agent: Mutt/1.5.23 (2014-03-12)
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev]
 =?utf-8?b?562U5aSNOiAgW1BBVENIXSBBZGQgdXNlciBkZWZpbmVk? =?utf-8?q?_tag_calculation_callback_tolibrte_distributor.?X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 07 Nov 2014 14:34:52 -0000

On Fri, Nov 07, 2014 at 04:31:18PM +0200, jigsaw wrote:
> Hi Bruce,
>
> Pls have a quick look at the diff to see if this is exactly what you mean
> about the bitmask.
> I just wrote it without even compiling, just to express the idea. So it may
> leave some places unpatched.
> If this is agreed, I will make a decent test to verify it before sending
> the patch for RFC.
>
> diff --git a/lib/librte_distributor/rte_distributor.c
> b/lib/librte_distributor/rte_di
> index 585ff88..d606bcf 100644
> --- a/lib/librte_distributor/rte_distributor.c
> +++ b/lib/librte_distributor/rte_distributor.c
> @@ -92,6 +92,8 @@ struct rte_distributor {
>         unsigned num_workers;                 /**< Number of workers
> polling */
>
>         uint32_t in_flight_tags[RTE_MAX_LCORE];
> +       uint32_t in_flight_bitmask;
> +
>         struct rte_distributor_backlog backlog[RTE_MAX_LCORE];
>
>         union rte_distributor_buffer bufs[RTE_MAX_LCORE];
> @@ -188,6 +190,7 @@ static inline void
>  handle_worker_shutdown(struct rte_distributor *d, unsigned wkr)
>  {
>         d->in_flight_tags[wkr] = 0;
> +       d->in_flight_mask &= ~(1 << wkr);
>         d->bufs[wkr].bufptr64 = 0;
>         if (unlikely(d->backlog[wkr].count != 0)) {
>                 /* On return of a packet, we need to move the
> @@ -241,6 +244,7 @@ process_returns(struct rte_distributor *d)
>                         else {
>                                 d->bufs[wkr].bufptr64 = RTE_DISTRIB_GET_BUF;
>                                 d->in_flight_tags[wkr] = 0;
> +                               d->in_flight_mask &= ~(1 << wkr);
>                         }
>                         oldbuf = data >> RTE_DISTRIB_FLAG_BITS;
>                 } else if (data & RTE_DISTRIB_RETURN_BUF) {
> @@ -282,12 +286,13 @@ rte_distributor_process(struct rte_distributor *d,
>                         next_mb = mbufs[next_idx++];
>                         next_value = (((int64_t)(uintptr_t)next_mb)
>                                         << RTE_DISTRIB_FLAG_BITS);
> -                       new_tag = (next_mb->hash.rss | 1);
> +                       new_tag = next_mb->hash.rss;
>
>                         uint32_t match = 0;
>                         unsigned i;
>                         for (i = 0; i < d->num_workers; i++)
> -                               match |= (!(d->in_flight_tags[i] ^ new_tag)
> +                               match |= (((!(d->in_flight_tags[i] ^
> new_tag)) &
> +                                               (d->in_flight_bitmask >> i))

I would not do the bitmask comparison here, as that's extra instruction in the
loop. Instead, because its a bitmask, build up the match variable as it was
before, and then just do a single and operation afterwards, outside the loop
body.

/Bruce

>                                         << i);
>
>                         if (match) {
> @@ -309,6 +314,7 @@ rte_distributor_process(struct rte_distributor *d,
>                         else {
>                                 d->bufs[wkr].bufptr64 = next_value;
>                                 d->in_flight_tags[wkr] = new_tag;
> +                               d->in_flight_bitmask |= 1 << wkr;
>                                 next_mb = NULL;
>                         }
>                         oldbuf = data >> RTE_DISTRIB_FLAG_BITS;
>
>
>