Hi Anatoly,
Thanks for the comments!
So now running with the correct initializations: first fork(), then call rte_eal_init() in the secondary processes.
When using hugepages, DPDK is up and running!
Then, tried to use the external memory:
- On primary: followed the example of create_extmem() in testpmd.c [without calling calc_mem_size() & alloc_mem(), as we already have virtual addresses and sizes from previous mmap() to the external memory regions].
Unfortunately, calling rte_mem_virt2iova() [which calls rte_mem_virt2phy()] always returns RTE_BAD_IOVA.
- On secondaries, rte_eal_init() successfully finished, but then calling rte_mempool_lookup() gets null data...
☹
Do you have any idea?
Thanks,
Asaf
-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com>
Sent: Tuesday, July 12, 2022 16:14
To: Asaf Sinai <AsafSi@Radware.com>; dev@dpdk.org
Subject: Re: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes
Hi Asaf,
On 12-Jul-22 7:05 AM, Asaf Sinai wrote:
> Hi,
>
> We run DPDK 19.11.3 with multi processes.
>
> When using external memory for DPDK (instead of huge pages), we are
> unable to receive traffic in the secondary processes.
>
> * We have external physical memory regions (excluded from Linux):
>
> /[ULP-NG]# cat /proc/cmdline/
>
> /console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0
> rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0/
>
> /memmap=0x1000000!0x60000000 *memmap=0x90000000!0x800000000
> memmap=0x90000000!0x1800000000*quiet cloud-init=disabled/
>
> * We make all DPDK initializations in the primary process, including
> mapping the mentioned memory regions via “/dev/mem”.
>
> After these memory mapping, we register the external memory, as
> described in
>
http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#suppo
> rt-for-externally-allocated-memory
> <http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#supp
> ort-for-externally-allocated-memory>,
> and allocate memory pools:
>
> /.../
>
> //* Map physical to virtual *//
>
> /int memFd = open("/dev/mem", O_RDWR);/
>
> /extMemInfo[i].pageSize = pPagesInfo->maxPageSize;/
>
> /extMemInfo[i].memRegSize = 0x0000000080000000;/
>
> /extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize,
> PROT_READ | PROT_WRITE,/
>
> / MAP_SHARED, memFd, memRegPhysAddr[i]);/
>
> //
>
> //* Create heap *//
>
> /sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);/
>
> /rv = rte_malloc_heap_create(extMemInfo[i].heapName);/
>
> //
>
> //* Save heap socket ID *//
>
> /rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);/
>
> /extMemInfo[i].socketId = rv;/
>
> //
>
> //* Add memory region to heap *//
>
> /rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName,
> extMemInfo[i].memRegAddr,/
>
> / extMemInfo[i].memRegSize, NULL, 0,
> extMemInfo[i].pageSize);/
>
> /.../
Please correct me if I'm wrong, but it seems like you're specifying NULL as IOVA table, so that means external memory will not get IOVA addresses set. You need to find (probably using rte_mem_virt2phys() function) the individual page
addresses, put them into an array, and pass it as an argument to add_memory call. See documentation[1], particularly the iova_addrs[] parameter.
[1]
http://doc.dpdk.org/api/rte__malloc_8h.html#af6360dea35bdf162feeb2b62cf149fd3
>
> //* Allocate memory pool *//
>
> /memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE,
> poolCacheSize,/
>
> / sizeof(struct rte_pktmbuf_pool_private),/
>
> / rte_pktmbuf_pool_init, NULL,/
>
> / und_pktmbuf_init, NULL,
> extMemInfo[i].socketId, DPDK_NO_FLAGS);/
>
> /.../
>
> Please note, that during calls to “/rte_malloc_heap_memory_add/”, we see
> the following warnings:
>
> */EAL: WARNING! Base virtual address hint (0x2101005000 !=
> 0x7ffff4627000) not respected!/*
>
> /EAL: This may cause issues with mapping memory into secondary processes/
>
> */EAL: WARNING! Base virtual address hint (0x210100b000 !=
> 0x7ffff4619000) not respected!/*
>
> /EAL: This may cause issues with mapping memory into secondary processes/
>
> And after executing “/rte_mempool_create/”, physical addresses of the
> allocated memory zones are bad:
Yes, because you did not specify them as part of
rte_malloc_heap_add_memory() call :)
>
> /[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0,
> socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104,
> pageSize=2097152/
>
> /[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256,
> vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0,
> *paddr=0xffffffffffffffff-0x1840ff*, len=1589504, hugepage_sz=2MB/
>
> * In the next step, we spawn the secondary processes from the primary
> one, using fork().
>
> This way, all DPDK data and memory mappings are the same on all
> processes. For example:
This is a recipe for disaster, because DPDK creates internal data
structures that are mapped as files, and primary process expects to be
the sole owner and user of those data structures. If you would like to
create a secondary process using fork() at all, you should fork() first,
and then call rte_eal_init() in the secondary process. A forked primary
process is not what we mean when we say "secondary process".
In addition, see documentation[1] for `rte_malloc_heap_memory_add()` -
there's a note there that states:
Before accessing this memory in other processes, it needs to be attached
in each of those processes by calling rte_malloc_heap_memory_attach in
each other process.
So, not only you need to call secondary process EAL init, you will also
have to call `rte_malloc_heap_memory_attach()` before you can use your
shared external memory in your secondary process.
Hope this is helpful!
>
> _Mapping in primary process:_
>
> /cat /proc/1875/maps/
>
> /.../
>
> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06
> 5 /dev/mem/*
>
> */7fff58000000-7fffe8000000 rw-s 800000000 00:06
> 5 /dev/mem/*
>
> /.../
>
> _Mapping in secondary process:_
>
> /cat /proc/2676/maps/
>
> /.../
>
> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06
> 5 /dev/mem/*
>
> */7fff58000000-7fffe8000000 rw-s 800000000 00:06
> 5 /dev/mem/*
>
> /.../
>
> * Unfortunately, when traffic is received by the secondary processes,
> we see the following printout on the primary process:
>
> */i40e_dev_alarm_handler(): ICR0: malicious programming detected/*
>
> * Additionally, we saw no example of using external physical shared
> memory on secondary processes.
>
> DPDK test applications show usage of anonymous private memory on the
> primary process only.
>
> What are we missing?
>
> Can you please advise?
>
> Thanks,
>
> Asaf
>
> Radware
>
--
Thanks,
Anatoly