RE: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes

[Asaf Sinai <AsafSi@Radware.com>]

[-- Attachment #1: Type: text/plain, Size: 8299 bytes --]

As calling rte_mem_virt2iova() returns RTE_BAD_IOVA, tried to supply the correct matching physical addresses in iova_addrs[], but it did not help either: secondary processes still get null data in rte_mempool_lookup().
How does secondary process get info from primary? Via files in /run/dpdk/rte/ ?

From: Asaf Sinai <AsafSi@Radware.com>
Sent: Thursday, July 14, 2022 13:25
To: Burakov, Anatoly <anatoly.burakov@intel.com>; dev@dpdk.org
Subject: RE: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes


Hi Anatoly,



Thanks for the comments!

So now running with the correct initializations: first fork(), then call rte_eal_init() in the secondary processes.

When using hugepages, DPDK is up and running!



Then, tried to use the external memory:

- On primary: followed the example of create_extmem() in testpmd.c [without calling calc_mem_size() & alloc_mem(), as we already have virtual addresses and sizes from previous mmap() to the external memory regions].

  Unfortunately, calling rte_mem_virt2iova() [which calls rte_mem_virt2phy()] always returns RTE_BAD_IOVA.

- On secondaries, rte_eal_init() successfully finished, but then calling rte_mempool_lookup() gets null data... ☹



Do you have any idea?



Thanks,

Asaf



-----Original Message-----
From: Burakov, Anatoly <anatoly.burakov@intel.com<mailto:anatoly.burakov@intel.com>>
Sent: Tuesday, July 12, 2022 16:14
To: Asaf Sinai <AsafSi@Radware.com<mailto:AsafSi@Radware.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
Subject: Re: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes



Hi Asaf,



On 12-Jul-22 7:05 AM, Asaf Sinai wrote:

> Hi,

>

> We run DPDK 19.11.3 with multi processes.

>

> When using external memory for DPDK (instead of huge pages), we are

> unable to receive traffic in the secondary processes.

>

>   * We have external physical memory regions (excluded from Linux):

>

> /[ULP-NG]# cat /proc/cmdline/

>

> /console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0

> rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0/

>

> /memmap=0x1000000!0x60000000 *memmap=0x90000000!0x800000000

> memmap=0x90000000!0x1800000000*quiet cloud-init=disabled/

>

>   * We make all DPDK initializations in the primary process, including

>     mapping the mentioned memory regions via “/dev/mem”.

>

> After these memory mapping, we register the external memory, as

> described in

> http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#suppo

> rt-for-externally-allocated-memory

> <http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#supp

> ort-for-externally-allocated-memory>,

> and allocate memory pools:

>

> /.../

>

> //* Map physical to virtual *//

>

> /int memFd = open("/dev/mem", O_RDWR);/

>

> /extMemInfo[i].pageSize   = pPagesInfo->maxPageSize;/

>

> /extMemInfo[i].memRegSize = 0x0000000080000000;/

>

> /extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize,

> PROT_READ | PROT_WRITE,/

>

> /                             MAP_SHARED, memFd, memRegPhysAddr[i]);/

>

> //

>

> //* Create heap *//

>

> /sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);/

>

> /rv = rte_malloc_heap_create(extMemInfo[i].heapName);/

>

> //

>

> //* Save heap socket ID *//

>

> /rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);/

>

> /extMemInfo[i].socketId = rv;/

>

> //

>

> //* Add memory region to heap *//

>

> /rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName,

> extMemInfo[i].memRegAddr,/

>

> /                                extMemInfo[i].memRegSize, NULL, 0,

> extMemInfo[i].pageSize);/

>

> /.../



Please correct me if I'm wrong, but it seems like you're specifying NULL as IOVA table, so that means external memory will not get IOVA addresses set. You need to find (probably using rte_mem_virt2phys() function) the individual page addresses, put them into an array, and pass it as an argument to add_memory call. See documentation[1], particularly the iova_addrs[] parameter.



[1]

http://doc.dpdk.org/api/rte__malloc_8h.html#af6360dea35bdf162feeb2b62cf149fd3



>

> //* Allocate memory pool *//

>

> /memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE,

> poolCacheSize,/

>

> /                             sizeof(struct rte_pktmbuf_pool_private),/

>

> /                             rte_pktmbuf_pool_init, NULL,/

>

> /                             und_pktmbuf_init, NULL,

> extMemInfo[i].socketId, DPDK_NO_FLAGS);/

>

> /.../

>

> Please note, that during calls to “/rte_malloc_heap_memory_add/”, we see

> the following warnings:

>

> */EAL: WARNING! Base virtual address hint (0x2101005000 !=

> 0x7ffff4627000) not respected!/*

>

> /EAL:    This may cause issues with mapping memory into secondary processes/

>

> */EAL: WARNING! Base virtual address hint (0x210100b000 !=

> 0x7ffff4619000) not respected!/*

>

> /EAL:    This may cause issues with mapping memory into secondary processes/

>

> And after executing “/rte_mempool_create/”, physical addresses of the

> allocated memory zones are bad:



Yes, because you did not specify them as part of

rte_malloc_heap_add_memory() call :)



>

> /[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0,

> socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104,

> pageSize=2097152/

>

> /[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256,

> vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0,

> *paddr=0xffffffffffffffff-0x1840ff*, len=1589504, hugepage_sz=2MB/

>

>   * In the next step, we spawn the secondary processes from the primary

>     one, using fork().

>

> This way, all DPDK data and memory mappings are the same on all

> processes. For example:



This is a recipe for disaster, because DPDK creates internal data

structures that are mapped as files, and primary process expects to be

the sole owner and user of those data structures. If you would like to

create a secondary process using fork() at all, you should fork() first,

and then call rte_eal_init() in the secondary process. A forked primary

process is not what we mean when we say "secondary process".



In addition, see documentation[1] for `rte_malloc_heap_memory_add()` -

there's a note there that states:



Before accessing this memory in other processes, it needs to be attached

in each of those processes by calling rte_malloc_heap_memory_attach in

each other process.



So, not only you need to call secondary process EAL init, you will also

have to call `rte_malloc_heap_memory_attach()` before you can use your

shared external memory in your secondary process.



Hope this is helpful!



>

> _Mapping in primary process:_

>

> /cat /proc/1875/maps/

>

> /.../

>

> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06

> 5                        /dev/mem/*

>

> */7fff58000000-7fffe8000000 rw-s 800000000 00:06

> 5                         /dev/mem/*

>

> /.../

>

> _Mapping in secondary process:_

>

> /cat /proc/2676/maps/

>

> /.../

>

> */7ffec8000000-7fff58000000 rw-s 1800000000 00:06

> 5                        /dev/mem/*

>

> */7fff58000000-7fffe8000000 rw-s 800000000 00:06

> 5                         /dev/mem/*

>

> /.../

>

>   * Unfortunately, when traffic is received by the secondary processes,

>     we see the following printout on the primary process:

>

> */i40e_dev_alarm_handler(): ICR0: malicious programming detected/*

>

>   * Additionally, we saw no example of using external physical shared

>     memory on secondary processes.

>

> DPDK test applications show usage of anonymous private memory on the

> primary process only.

>

> What are we missing?

>

> Can you please advise?

>

> Thanks,

>

> Asaf

>

> Radware

>





--

Thanks,

Anatoly

[-- Attachment #2: Type: text/html, Size: 21477 bytes --]