[dpdk-dev] Multiprocess /dev/uio mmap Bug/possible fix

[dpdk-dev] Multiprocess /dev/uio mmap Bug/possible fix

[Stefan Baranoff]

All,

If this was fixed in 1.7 and I missed it I apologize (but it looks from
source to still be broken). I am using DPDK 1.6.0r2 (will be upgrading to
1.7.0 soon) on RHEL 6.4. I've converted the functions below to 1.7.0
names/locations since it looks to still be an issue there.

tl;dr -- to get examples/symmetric_mp to work at all I had to manually
force the address used to map PCI UIO devices in the primary process which
I achieved by:
In eal_memory.c :: get_virtual_area(...) -- Use directly then increment
internal_config.base_virtaddr instead of incrementing/adding baseaddr_offset
In eal_pci_uio.c in 1.7.0 :: pci_uio_map_resource -- In the primary
process, use and increment internal_config.base_virtaddr as the first
argument when calling pci_uio_map_resource (instead of NULL).


Background/reason:
Recently I was trying to get multiprocessing with each process reading from
one queue working and ran into issues with the call to mmap the memory for
/dev/uioX. Even the example 'symmetric_mp' application was failing. The
issue is that there are two memory regions which are mmap'd for each
/dev/uioX device and the second one, in the secondary process, is being
mapped to an address other than the requested address. This causes the
check for pci_map_resource(...) != uio_res->maps[i].addr to fail. I tracked
the reason for not using the requested address to the mmap call to create
the stack for a thread created during one of (I forget which - I think eal)
eal/pci/pmd_init calls which is used to handle interrupts.

With DPDK compiled as a single shared library on our system the memory
address handed for the second memory reach for each /dev/uioX in the
primary process is right in the middle of the stack for that thread.

My possible fix: use --base-virtaddr to populate the requested address for
mmaping the memory for /dev/uioX
I am not sure if this is safe but if it is I can submit a patch. My
approach was in eal_memory.c to, rather than keep an 'offset' variable
locally just increment the global base_virtaddr value each time through
that loop and then use the resulting final value when doing the /dev/uioX
mappings.

Of course in 1.6.0r2 I also ran into the errno bug with parsing
--base-virtaddr and strtoull but it looks like that was fixed in 1.7.0.


Thoughts? Thanks!
Stefan

Re: [dpdk-dev] Multiprocess /dev/uio mmap Bug/possible fix

[Gray, Mark D]

> All,
> 
> If this was fixed in 1.7 and I missed it I apologize (but it looks from source to
> still be broken). I am using DPDK 1.6.0r2 (will be upgrading to
> 1.7.0 soon) on RHEL 6.4. I've converted the functions below to 1.7.0
> names/locations since it looks to still be an issue there.
> 
> tl;dr -- to get examples/symmetric_mp to work at all I had to manually force
> the address used to map PCI UIO devices in the primary process which I
> achieved by:
> In eal_memory.c :: get_virtual_area(...) -- Use directly then increment
> internal_config.base_virtaddr instead of incrementing/adding
> baseaddr_offset In eal_pci_uio.c in 1.7.0 :: pci_uio_map_resource -- In the
> primary process, use and increment internal_config.base_virtaddr as the first
> argument when calling pci_uio_map_resource (instead of NULL).
> 
> 
> Background/reason:
> Recently I was trying to get multiprocessing with each process reading from
> one queue working and ran into issues with the call to mmap the memory for
> /dev/uioX. Even the example 'symmetric_mp' application was failing. The
> issue is that there are two memory regions which are mmap'd for each
> /dev/uioX device and the second one, in the secondary process, is being
> mapped to an address other than the requested address. This causes the
> check for pci_map_resource(...) != uio_res->maps[i].addr to fail. I tracked
> the reason for not using the requested address to the mmap call to create
> the stack for a thread created during one of (I forget which - I think eal)
> eal/pci/pmd_init calls which is used to handle interrupts.
> 
> With DPDK compiled as a single shared library on our system the memory
> address handed for the second memory reach for each /dev/uioX in the
> primary process is right in the middle of the stack for that thread.
> 
> My possible fix: use --base-virtaddr to populate the requested address for
> mmaping the memory for /dev/uioX I am not sure if this is safe but if it is I
> can submit a patch. My approach was in eal_memory.c to, rather than keep
> an 'offset' variable locally just increment the global base_virtaddr value each
> time through that loop and then use the resulting final value when doing the
> /dev/uioX mappings.
> 
> Of course in 1.6.0r2 I also ran into the errno bug with parsing --base-virtaddr
> and strtoull but it looks like that was fixed in 1.7.0.

We also see this. There is a patch in a patchset that resolves this

http://dpdk.org/ml/archives/dev/2014-June/003689.html

It hasn’t been pushed yet.

> 
> 
> Thoughts? Thanks!
> Stefan