* [dpdk-dev] Using DPDK in a multiprocess environment @ 2014-04-08 15:12 Etai Lev Ran 2014-04-08 15:50 ` Shaw, Jeffrey B 0 siblings, 1 reply; 6+ messages in thread From: Etai Lev Ran @ 2014-04-08 15:12 UTC (permalink / raw) To: dev Hi, I'd like to split DPDK application functionality into a setup (primary) process and business logic (secondary) processes. The secondary processes access the hardware queues directly (exclusive queue per process) and not through software rings. I'm running into an initialization problem: - The primary starts and sets up memory and ports and then goes to sleep waiting for termination signal - Secondary processes fail when probing the PCI bus for devices (required, otherwise I get 0 ports visible in the secondary) The error is directly related to the secondary failing to get the *same* virtual address for mmap'ing the UIO device fd's. The reason is that the secondary processes has considerably more shared objects loaded and some of these are loaded and mapped into addresses which the primary used to map UIO fd's. The pci_map_resource() (linuxapp/eal_pci.c) code explicitly requires that the secondary processes get the same mmap'ed address as given to the primary. 1) Is this behavior (same mmap address) required? 2) If so, is there a workaround to cause PCI areas of UIO devices to be mapped to the same location in arbitrary processes? The samples work just fine since all primary and secondary processes have similar set and load order for .so's Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR disabled. Thanks, Etai ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Using DPDK in a multiprocess environment 2014-04-08 15:12 [dpdk-dev] Using DPDK in a multiprocess environment Etai Lev Ran @ 2014-04-08 15:50 ` Shaw, Jeffrey B 2014-04-08 16:07 ` elevran 0 siblings, 1 reply; 6+ messages in thread From: Shaw, Jeffrey B @ 2014-04-08 15:50 UTC (permalink / raw) To: Etai Lev Ran, dev Have you tried calling "rte_eal_init()" closer to the beginning of the program in your secondary process (i.e. the first thing in main())? The same mmap address is required. The reason is simple, if process A thinks the virtual address of an mbuf is 123, and process B thinks the virtual address of the same mbuf is 456, either process may segmentation fault, accessing mbuf memory that is not actually mapped into the processes address space. Jeff -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran Sent: Tuesday, April 08, 2014 8:13 AM To: dev@dpdk.org Subject: [dpdk-dev] Using DPDK in a multiprocess environment Hi, I'd like to split DPDK application functionality into a setup (primary) process and business logic (secondary) processes. The secondary processes access the hardware queues directly (exclusive queue per process) and not through software rings. I'm running into an initialization problem: - The primary starts and sets up memory and ports and then goes to sleep waiting for termination signal - Secondary processes fail when probing the PCI bus for devices (required, otherwise I get 0 ports visible in the secondary) The error is directly related to the secondary failing to get the *same* virtual address for mmap'ing the UIO device fd's. The reason is that the secondary processes has considerably more shared objects loaded and some of these are loaded and mapped into addresses which the primary used to map UIO fd's. The pci_map_resource() (linuxapp/eal_pci.c) code explicitly requires that the secondary processes get the same mmap'ed address as given to the primary. 1) Is this behavior (same mmap address) required? 2) If so, is there a workaround to cause PCI areas of UIO devices to be mapped to the same location in arbitrary processes? The samples work just fine since all primary and secondary processes have similar set and load order for .so's Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR disabled. Thanks, Etai ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Using DPDK in a multiprocess environment 2014-04-08 15:50 ` Shaw, Jeffrey B @ 2014-04-08 16:07 ` elevran 2014-04-08 17:00 ` Rogers, Gerald 0 siblings, 1 reply; 6+ messages in thread From: elevran @ 2014-04-08 16:07 UTC (permalink / raw) To: Shaw, Jeffrey B; +Cc: dev Jeff, Thanks for the quick reply. I'll see if calling eal_init earlier resolves the problem I'm seeing. I'm not sure this will resolve the issue if shared objects are loaded before main() starts... I understand the rationale for having the same mbuf addresses across processes. And indeed they're mapped just fine (--virt-addr also gives some control over the mapping?). I was wondering if the same logic applies to the mapping of device PCI addresses. Are they shared or passed around between processes in the same way? Thanks again for the quick response, Etai בתאריך 8 באפר 2014 18:54, "Shaw, Jeffrey B" <jeffrey.b.shaw@intel.com> כתב: > Have you tried calling "rte_eal_init()" closer to the beginning of the > program in your secondary process (i.e. the first thing in main())? > > The same mmap address is required. The reason is simple, if process A > thinks the virtual address of an mbuf is 123, and process B thinks the > virtual address of the same mbuf is 456, either process may segmentation > fault, accessing mbuf memory that is not actually mapped into the processes > address space. > > Jeff > > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran > Sent: Tuesday, April 08, 2014 8:13 AM > To: dev@dpdk.org > Subject: [dpdk-dev] Using DPDK in a multiprocess environment > > Hi, > > > > I'd like to split DPDK application functionality into a setup (primary) > process and business logic (secondary) processes. > > The secondary processes access the hardware queues directly (exclusive > queue per process) and not through software rings. > > > > I'm running into an initialization problem: > > - The primary starts and sets up memory and ports and then goes to > sleep waiting for termination signal > > - Secondary processes fail when probing the PCI bus for devices > (required, otherwise I get 0 ports visible in the secondary) > > > > The error is directly related to the secondary failing to get the *same* > virtual address for mmap'ing the UIO device fd's. > > The reason is that the secondary processes has considerably more shared > objects loaded and some of these are > > loaded and mapped into addresses which the primary used to map UIO fd's. > > The pci_map_resource() (linuxapp/eal_pci.c) code explicitly requires that > the secondary processes get the same mmap'ed > > address as given to the primary. > > > > 1) Is this behavior (same mmap address) required? > > 2) If so, is there a workaround to cause PCI areas of UIO devices to > be > mapped to the same location in arbitrary processes? > > > > The samples work just fine since all primary and secondary processes have > similar set and load order for .so's > > > > Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR > disabled. > > > > Thanks, > > Etai > > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Using DPDK in a multiprocess environment 2014-04-08 16:07 ` elevran @ 2014-04-08 17:00 ` Rogers, Gerald 2014-04-09 9:25 ` Richardson, Bruce 0 siblings, 1 reply; 6+ messages in thread From: Rogers, Gerald @ 2014-04-08 17:00 UTC (permalink / raw) To: elevran, Shaw, Jeffrey B; +Cc: dev Etai, If this doesn’t work, then you will need to change the virtual address range that is used by DPDK. By default this is set dynamically, however; with DPDK 1.6you can change it to any region in the virtual address space you want. The problem you have is what you stated, the secondary process is built with more shared libraries, which load upon application start, and are occupying the region that DPDK allocates in the primary for shared regions. In DPDK version 1.6 there is an option to change the base address. It is --base-virtaddr With this option you can set the base address for where the huge pages are mapped into the process virtual address space. This is all implemented within $DPDK_DIR/lib/librte_eal/linuxapp/eal/eal_memory.c Gerald On 4/8/14, 9:07 AM, "elevran" <elevran@gmail.com> wrote: >Jeff, > >Thanks for the quick reply. > >I'll see if calling eal_init earlier resolves the problem I'm seeing. I'm >not sure this will resolve the issue if shared objects are loaded before >main() starts... > >I understand the rationale for having the same mbuf addresses across >processes. And indeed they're mapped just fine (--virt-addr also gives >some >control over the mapping?). >I was wondering if the same logic applies to the mapping of device PCI >addresses. Are they shared or passed around between processes in the same >way? > >Thanks again for the quick response, >Etai >בתאריך 8 באפר 2014 18:54, "Shaw, Jeffrey B" <jeffrey.b.shaw@intel.com> >כתב: > >> Have you tried calling "rte_eal_init()" closer to the beginning of the >> program in your secondary process (i.e. the first thing in main())? >> >> The same mmap address is required. The reason is simple, if process A >> thinks the virtual address of an mbuf is 123, and process B thinks the >> virtual address of the same mbuf is 456, either process may segmentation >> fault, accessing mbuf memory that is not actually mapped into the >>processes >> address space. >> >> Jeff >> >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran >> Sent: Tuesday, April 08, 2014 8:13 AM >> To: dev@dpdk.org >> Subject: [dpdk-dev] Using DPDK in a multiprocess environment >> >> Hi, >> >> >> >> I'd like to split DPDK application functionality into a setup (primary) >> process and business logic (secondary) processes. >> >> The secondary processes access the hardware queues directly (exclusive >> queue per process) and not through software rings. >> >> >> >> I'm running into an initialization problem: >> >> - The primary starts and sets up memory and ports and then >>goes to >> sleep waiting for termination signal >> >> - Secondary processes fail when probing the PCI bus for devices >> (required, otherwise I get 0 ports visible in the secondary) >> >> >> >> The error is directly related to the secondary failing to get the *same* >> virtual address for mmap'ing the UIO device fd's. >> >> The reason is that the secondary processes has considerably more shared >> objects loaded and some of these are >> >> loaded and mapped into addresses which the primary used to map UIO fd's. >> >> The pci_map_resource() (linuxapp/eal_pci.c) code explicitly requires >>that >> the secondary processes get the same mmap'ed >> >> address as given to the primary. >> >> >> >> 1) Is this behavior (same mmap address) required? >> >> 2) If so, is there a workaround to cause PCI areas of UIO devices >>to >> be >> mapped to the same location in arbitrary processes? >> >> >> >> The samples work just fine since all primary and secondary processes >>have >> similar set and load order for .so's >> >> >> >> Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR >> disabled. >> >> >> >> Thanks, >> >> Etai >> >> >> >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Using DPDK in a multiprocess environment 2014-04-08 17:00 ` Rogers, Gerald @ 2014-04-09 9:25 ` Richardson, Bruce 2014-04-10 10:26 ` Etai Lev Ran 0 siblings, 1 reply; 6+ messages in thread From: Richardson, Bruce @ 2014-04-09 9:25 UTC (permalink / raw) To: Rogers, Gerald, elevran, Shaw, Jeffrey B; +Cc: dev As a plan B (or C, or D, etc.) you could also try linking your primary process against those same shared libraries, even if they are unused by it. Hopefully that may have the same effect in the primary as in the secondary processes of adjusting your address space region and allow things to get mapped properly. /Bruce > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rogers, Gerald > Sent: Tuesday, April 08, 2014 6:00 PM > To: elevran; Shaw, Jeffrey B > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Using DPDK in a multiprocess environment > > Etai, > > If this doesn’t work, then you will need to change the virtual address range that > is used by DPDK. By default this is set dynamically, however; with DPDK 1.6you > can change it to any region in the virtual address space you want. > > The problem you have is what you stated, the secondary process is built with > more shared libraries, which load upon application start, and are occupying the > region that DPDK allocates in the primary for shared regions. > > In DPDK version 1.6 there is an option to change the base address. It is --base- > virtaddr > > With this option you can set the base address for where the huge pages are > mapped into the process virtual address space. > > This is all implemented within > $DPDK_DIR/lib/librte_eal/linuxapp/eal/eal_memory.c > > Gerald > > > > > > On 4/8/14, 9:07 AM, "elevran" <elevran@gmail.com> wrote: > > >Jeff, > > > >Thanks for the quick reply. > > > >I'll see if calling eal_init earlier resolves the problem I'm seeing. > >I'm not sure this will resolve the issue if shared objects are loaded > >before > >main() starts... > > > >I understand the rationale for having the same mbuf addresses across > >processes. And indeed they're mapped just fine (--virt-addr also gives > >some control over the mapping?). > >I was wondering if the same logic applies to the mapping of device PCI > >addresses. Are they shared or passed around between processes in the > >same way? > > > >Thanks again for the quick response, > >Etai > >בתאריך 8 באפר 2014 18:54, "Shaw, Jeffrey B" <jeffrey.b.shaw@intel.com> > >כתב: > > > >> Have you tried calling "rte_eal_init()" closer to the beginning of > >> the program in your secondary process (i.e. the first thing in main())? > >> > >> The same mmap address is required. The reason is simple, if process > >>A thinks the virtual address of an mbuf is 123, and process B thinks > >>the virtual address of the same mbuf is 456, either process may > >>segmentation fault, accessing mbuf memory that is not actually mapped > >>into the processes address space. > >> > >> Jeff > >> > >> -----Original Message----- > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran > >> Sent: Tuesday, April 08, 2014 8:13 AM > >> To: dev@dpdk.org > >> Subject: [dpdk-dev] Using DPDK in a multiprocess environment > >> > >> Hi, > >> > >> > >> > >> I'd like to split DPDK application functionality into a setup > >> (primary) process and business logic (secondary) processes. > >> > >> The secondary processes access the hardware queues directly > >> (exclusive queue per process) and not through software rings. > >> > >> > >> > >> I'm running into an initialization problem: > >> > >> - The primary starts and sets up memory and ports and then > >>goes to > >> sleep waiting for termination signal > >> > >> - Secondary processes fail when probing the PCI bus for devices > >> (required, otherwise I get 0 ports visible in the secondary) > >> > >> > >> > >> The error is directly related to the secondary failing to get the > >> *same* virtual address for mmap'ing the UIO device fd's. > >> > >> The reason is that the secondary processes has considerably more > >> shared objects loaded and some of these are > >> > >> loaded and mapped into addresses which the primary used to map UIO fd's. > >> > >> The pci_map_resource() (linuxapp/eal_pci.c) code explicitly requires > >>that the secondary processes get the same mmap'ed > >> > >> address as given to the primary. > >> > >> > >> > >> 1) Is this behavior (same mmap address) required? > >> > >> 2) If so, is there a workaround to cause PCI areas of UIO devices > >>to > >> be > >> mapped to the same location in arbitrary processes? > >> > >> > >> > >> The samples work just fine since all primary and secondary processes > >>have similar set and load order for .so's > >> > >> > >> > >> Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR > >> disabled. > >> > >> > >> > >> Thanks, > >> > >> Etai > >> > >> > >> > >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] Using DPDK in a multiprocess environment 2014-04-09 9:25 ` Richardson, Bruce @ 2014-04-10 10:26 ` Etai Lev Ran 0 siblings, 0 replies; 6+ messages in thread From: Etai Lev Ran @ 2014-04-10 10:26 UTC (permalink / raw) To: 'Richardson, Bruce', 'Rogers, Gerald', 'Shaw, Jeffrey B' Cc: dev Thanks, Bruce. Yes - artificial linking may be a viable workaround in some cases. However, in the general case, it seems that : a) multi-process DPDK applications work best when using a single (primary) process feeding secondary processes via SW rings; This requires a matching map of the shared area (huge pages); b) to allow multiple processes to access the HW directly (with exclusive queue assignment, though), the shared memory and PCI mapping must be the same in all processes, implying that they should be as similar as possible (e.g., *before* initializing the PCI resources they must load the same objects and map the same files in the same order) Deviations from above may result in an inoperable system due to mismatches in the memory maps. I think DPDK was designed mostly with use-case (a) above in mind (software rings), but that has the unfortunate downside of dedicating CPU core(s) for HW access. Regards, Etai -----Original Message----- From: Richardson, Bruce [mailto:bruce.richardson@intel.com] Sent: Wednesday, April 9, 2014 12:25 PM To: Rogers, Gerald; elevran; Shaw, Jeffrey B Cc: dev@dpdk.org Subject: RE: [dpdk-dev] Using DPDK in a multiprocess environment As a plan B (or C, or D, etc.) you could also try linking your primary process against those same shared libraries, even if they are unused by it. Hopefully that may have the same effect in the primary as in the secondary processes of adjusting your address space region and allow things to get mapped properly. /Bruce > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Rogers, Gerald > Sent: Tuesday, April 08, 2014 6:00 PM > To: elevran; Shaw, Jeffrey B > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] Using DPDK in a multiprocess environment > > Etai, > > If this doesn’t work, then you will need to change the virtual address > range that is used by DPDK. By default this is set dynamically, > however; with DPDK 1.6you can change it to any region in the virtual address space you want. > > The problem you have is what you stated, the secondary process is > built with more shared libraries, which load upon application start, > and are occupying the region that DPDK allocates in the primary for shared regions. > > In DPDK version 1.6 there is an option to change the base address. It > is --base- virtaddr > > With this option you can set the base address for where the huge pages > are mapped into the process virtual address space. > > This is all implemented within > $DPDK_DIR/lib/librte_eal/linuxapp/eal/eal_memory.c > > Gerald > > > > > > On 4/8/14, 9:07 AM, "elevran" <elevran@gmail.com> wrote: > > >Jeff, > > > >Thanks for the quick reply. > > > >I'll see if calling eal_init earlier resolves the problem I'm seeing. > >I'm not sure this will resolve the issue if shared objects are loaded > >before > >main() starts... > > > >I understand the rationale for having the same mbuf addresses across > >processes. And indeed they're mapped just fine (--virt-addr also > >gives some control over the mapping?). > >I was wondering if the same logic applies to the mapping of device > >PCI addresses. Are they shared or passed around between processes in > >the same way? > > > >Thanks again for the quick response, > >Etai > >בתאריך 8 באפר 2014 18:54, "Shaw, Jeffrey B" > ><jeffrey.b.shaw@intel.com> > >כתב: > > > >> Have you tried calling "rte_eal_init()" closer to the beginning of > >> the program in your secondary process (i.e. the first thing in main())? > >> > >> The same mmap address is required. The reason is simple, if > >>process A thinks the virtual address of an mbuf is 123, and process > >>B thinks the virtual address of the same mbuf is 456, either > >>process may segmentation fault, accessing mbuf memory that is not > >>actually mapped into the processes address space. > >> > >> Jeff > >> > >> -----Original Message----- > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Etai Lev Ran > >> Sent: Tuesday, April 08, 2014 8:13 AM > >> To: dev@dpdk.org > >> Subject: [dpdk-dev] Using DPDK in a multiprocess environment > >> > >> Hi, > >> > >> > >> > >> I'd like to split DPDK application functionality into a setup > >> (primary) process and business logic (secondary) processes. > >> > >> The secondary processes access the hardware queues directly > >> (exclusive queue per process) and not through software rings. > >> > >> > >> > >> I'm running into an initialization problem: > >> > >> - The primary starts and sets up memory and ports and then > >>goes to > >> sleep waiting for termination signal > >> > >> - Secondary processes fail when probing the PCI bus for devices > >> (required, otherwise I get 0 ports visible in the secondary) > >> > >> > >> > >> The error is directly related to the secondary failing to get the > >> *same* virtual address for mmap'ing the UIO device fd's. > >> > >> The reason is that the secondary processes has considerably more > >> shared objects loaded and some of these are > >> > >> loaded and mapped into addresses which the primary used to map UIO fd's. > >> > >> The pci_map_resource() (linuxapp/eal_pci.c) code explicitly > >>requires that the secondary processes get the same mmap'ed > >> > >> address as given to the primary. > >> > >> > >> > >> 1) Is this behavior (same mmap address) required? > >> > >> 2) If so, is there a workaround to cause PCI areas of UIO devices > >>to > >> be > >> mapped to the same location in arbitrary processes? > >> > >> > >> > >> The samples work just fine since all primary and secondary > >>processes have similar set and load order for .so's > >> > >> > >> > >> Using v1.6 on Ubuntu 12.04 64b, ixgbe devices, 1GB hugepages, ASLR > >> disabled. > >> > >> > >> > >> Thanks, > >> > >> Etai > >> > >> > >> > >> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-04-10 10:24 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-04-08 15:12 [dpdk-dev] Using DPDK in a multiprocess environment Etai Lev Ran 2014-04-08 15:50 ` Shaw, Jeffrey B 2014-04-08 16:07 ` elevran 2014-04-08 17:00 ` Rogers, Gerald 2014-04-09 9:25 ` Richardson, Bruce 2014-04-10 10:26 ` Etai Lev Ran
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).