From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ee0-f52.google.com (mail-ee0-f52.google.com [74.125.83.52]) by dpdk.org (Postfix) with ESMTP id BEE15307 for ; Thu, 10 Apr 2014 12:24:40 +0200 (CEST) Received: by mail-ee0-f52.google.com with SMTP id e49so2871650eek.11 for ; Thu, 10 Apr 2014 03:26:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:references:in-reply-to:subject:date:message-id :mime-version:content-type:content-transfer-encoding:thread-index :content-language; bh=Fcg2U8zcDJ8APCbk6olrWGwcD1H48/ft4+U4nXM/FiU=; b=qpXxAFijJFX7NsXAsnoPDMhRh+LgDoK1KvaUhjvPIyUK2cbsTltGRB7V+fAHj4zEhH IkfYUSk63YlXLK86/unfxq4c/RBoNscO8iq9sTrIiz5+vmPKjw/Yw74NoDrS9rb1UHUy ckCZF1sXyv4PX9XR0EosCp2vEPwpRa7tEXUaU7xuaTfKs8bUDNbWUFGKAJM1UZdgJFgD MHOeZ2C60ODheEBrxDZSjPTik12ro0FSk693gd4p/5ladHAntsRVwh1oQ46iRwpq0bFz Pdaf6t8sVfeCpS+N2KIIEcM2AySxmlGMtuwQzy5U+Q5N0KsGo7LRt0wNSdcgzd3XwEVD ck9A== X-Received: by 10.14.199.8 with SMTP id w8mr163276een.94.1397125577504; Thu, 10 Apr 2014 03:26:17 -0700 (PDT) Received: from elrlaptop (bzq-79-178-110-161.red.bezeqint.net. [79.178.110.161]) by mx.google.com with ESMTPSA id 4sm8862572eeq.33.2014.04.10.03.26.15 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 10 Apr 2014 03:26:16 -0700 (PDT) From: "Etai Lev Ran" To: "'Richardson, Bruce'" , "'Rogers, Gerald'" , "'Shaw, Jeffrey B'" References: <032101cf533c$f871bb60$e9553220$@gmail.com> <4032A54B6BB5F04B8C08B6CFF08C59285540FFDA@FMSMSX103.amr.corp.intel.com> <59AF69C657FD0841A61C55336867B5B01A9F9C8A@IRSMSX103.ger.corp.intel.com> In-Reply-To: <59AF69C657FD0841A61C55336867B5B01A9F9C8A@IRSMSX103.ger.corp.intel.com> Date: Thu, 10 Apr 2014 13:26:25 +0300 Message-ID: <001b01cf54a7$54cff460$fe6fdd20$@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQH9gXH9gcgKXbnhovofe1OBoDJrVwMQUDLDAay3khMCLHljiwGghRkLmmn6OBA= Content-Language: en-us Cc: dev@dpdk.org Subject: Re: [dpdk-dev] Using DPDK in a multiprocess environment X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2014 10:24:41 -0000 Thanks, Bruce. Yes - artificial linking may be a viable workaround in some cases.=20 However, in the general case, it seems that : a) multi-process DPDK applications work best when using a single = (primary) process feeding secondary processes via SW rings;=20 This requires a matching map of the shared area (huge pages); b) to allow multiple processes to access the HW directly (with exclusive = queue assignment, though), the shared memory and=20 PCI mapping must be the same in all processes, implying that they = should be as similar as possible (e.g., *before* initializing=20 the PCI resources they must load the same objects and map the same = files in the same order) Deviations from above may result in an inoperable system due to = mismatches in the memory maps. I think DPDK was designed mostly with use-case (a) above in mind = (software rings), but that has the unfortunate downside of=20 dedicating CPU core(s) for HW access. Regards, Etai -----Original Message----- From: Richardson, Bruce [mailto:bruce.richardson@intel.com]=20 Sent: Wednesday, April 9, 2014 12:25 PM To: Rogers, Gerald; elevran; Shaw, Jeffrey B Cc: dev@dpdk.org Subject: RE: [dpdk-dev] Using DPDK in a multiprocess environment As a plan B (or C, or D, etc.) you could also try linking your primary = process against those same shared libraries, even if they are unused by = it. Hopefully that may have the same effect in the primary as in the = secondary processes of adjusting your address space region and allow = things to get mapped properly. /Bruce=20 > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rogers, Gerald > Sent: Tuesday, April 08, 2014 6:00 PM > To: elevran; Shaw, Jeffrey B > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Using DPDK in a multiprocess environment >=20 > Etai, >=20 > If this doesn=E2=80=99t work, then you will need to change the virtual = address=20 > range that is used by DPDK. By default this is set dynamically,=20 > however; with DPDK 1.6you can change it to any region in the virtual = address space you want. >=20 > The problem you have is what you stated, the secondary process is=20 > built with more shared libraries, which load upon application start,=20 > and are occupying the region that DPDK allocates in the primary for = shared regions. >=20 > In DPDK version 1.6 there is an option to change the base address. It = > is --base- virtaddr >=20 > With this option you can set the base address for where the huge pages = > are mapped into the process virtual address space. >=20 > This is all implemented within > $DPDK_DIR/lib/librte_eal/linuxapp/eal/eal_memory.c >=20 > Gerald >=20 >=20 >=20 >=20 >=20 > On 4/8/14, 9:07 AM, "elevran" wrote: >=20 > >Jeff, > > > >Thanks for the quick reply. > > > >I'll see if calling eal_init earlier resolves the problem I'm seeing. > >I'm not sure this will resolve the issue if shared objects are loaded = > >before > >main() starts... > > > >I understand the rationale for having the same mbuf addresses across=20 > >processes. And indeed they're mapped just fine (--virt-addr also=20 > >gives some control over the mapping?). > >I was wondering if the same logic applies to the mapping of device=20 > >PCI addresses. Are they shared or passed around between processes in=20 > >the same way? > > > >Thanks again for the quick response, > >Etai > >=D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 8 =D7=91=D7=90=D7=A4=D7=A8 2014 = 18:54, "Shaw, Jeffrey B"=20 > > > >=D7=9B=D7=AA=D7=91: > > > >> Have you tried calling "rte_eal_init()" closer to the beginning of=20 > >> the program in your secondary process (i.e. the first thing in = main())? > >> > >> The same mmap address is required. The reason is simple, if=20 > >>process A thinks the virtual address of an mbuf is 123, and process = > >>B thinks the virtual address of the same mbuf is 456, either=20 > >>process may segmentation fault, accessing mbuf memory that is not=20 > >>actually mapped into the processes address space. > >> > >> Jeff > >> > >> -----Original Message----- > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran > >> Sent: Tuesday, April 08, 2014 8:13 AM > >> To: dev@dpdk.org > >> Subject: [dpdk-dev] Using DPDK in a multiprocess environment > >> > >> Hi, > >> > >> > >> > >> I'd like to split DPDK application functionality into a setup > >> (primary) process and business logic (secondary) processes. > >> > >> The secondary processes access the hardware queues directly=20 > >> (exclusive queue per process) and not through software rings. > >> > >> > >> > >> I'm running into an initialization problem: > >> > >> - The primary starts and sets up memory and ports and then > >>goes to > >> sleep waiting for termination signal > >> > >> - Secondary processes fail when probing the PCI bus for = devices > >> (required, otherwise I get 0 ports visible in the secondary) > >> > >> > >> > >> The error is directly related to the secondary failing to get the > >> *same* virtual address for mmap'ing the UIO device fd's. > >> > >> The reason is that the secondary processes has considerably more=20 > >> shared objects loaded and some of these are > >> > >> loaded and mapped into addresses which the primary used to map UIO = fd's. > >> > >> The pci_map_resource() (linuxapp/eal_pci.c) code explicitly=20 > >>requires that the secondary processes get the same mmap'ed > >> > >> address as given to the primary. > >> > >> > >> > >> 1) Is this behavior (same mmap address) required? > >> > >> 2) If so, is there a workaround to cause PCI areas of UIO = devices > >>to > >> be > >> mapped to the same location in arbitrary processes? > >> > >> > >> > >> The samples work just fine since all primary and secondary=20 > >>processes have similar set and load order for .so's > >> > >> > >> > >> Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR = > >> disabled. > >> > >> > >> > >> Thanks, > >> > >> Etai > >> > >> > >> > >>