DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes

[Asaf Sinai <AsafSi@Radware.com>]

[-- Attachment #1: Type: text/plain, Size: 5146 bytes --]

Hi,

We run DPDK 19.11.3 with multi processes.
When using external memory for DPDK (instead of huge pages), we are unable to receive traffic in the secondary processes.


  *   We have external physical memory regions (excluded from Linux):

[ULP-NG]# cat /proc/cmdline
console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0 rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0
memmap=0x1000000!0x60000000 memmap=0x90000000!0x800000000 memmap=0x90000000!0x1800000000 quiet cloud-init=disabled


  *   We make all DPDK initializations in the primary process, including mapping the mentioned memory regions via "/dev/mem".

After these memory mapping, we register the external memory, as described in http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory, and allocate memory pools:



...

/* Map physical to virtual */

int memFd = open("/dev/mem", O_RDWR);

extMemInfo[i].pageSize   = pPagesInfo->maxPageSize;

extMemInfo[i].memRegSize = 0x0000000080000000;

extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize, PROT_READ | PROT_WRITE,

                             MAP_SHARED, memFd, memRegPhysAddr[i]);



/* Create heap */

sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);

rv = rte_malloc_heap_create(extMemInfo[i].heapName);



/* Save heap socket ID */

rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);

extMemInfo[i].socketId = rv;



/* Add memory region to heap */

rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName, extMemInfo[i].memRegAddr,

                                extMemInfo[i].memRegSize, NULL, 0, extMemInfo[i].pageSize);

...

/* Allocate memory pool */

memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE, poolCacheSize,

                             sizeof(struct rte_pktmbuf_pool_private),

                             rte_pktmbuf_pool_init, NULL,

                             und_pktmbuf_init, NULL, extMemInfo[i].socketId, DPDK_NO_FLAGS);

...



Please note, that during calls to "rte_malloc_heap_memory_add", we see the following warnings:


EAL: WARNING! Base virtual address hint (0x2101005000 != 0x7ffff4627000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x210100b000 != 0x7ffff4619000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes



And after executing "rte_mempool_create", physical addresses of the allocated memory zones are bad:


[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0, socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104, pageSize=2097152
[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256, vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0, paddr=0xffffffffffffffff-0x1840ff, len=1589504, hugepage_sz=2MB



  *   In the next step, we spawn the secondary processes from the primary one, using fork().

This way, all DPDK data and memory mappings are the same on all processes. For example:

Mapping in primary process:
cat /proc/1875/maps
...
7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5                        /dev/mem
7fff58000000-7fffe8000000 rw-s 800000000 00:06 5                         /dev/mem
...

Mapping in secondary process:
cat /proc/2676/maps
...
7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5                        /dev/mem
7fff58000000-7fffe8000000 rw-s 800000000 00:06 5                         /dev/mem
...


  *   Unfortunately, when traffic is received by the secondary processes, we see the following printout on the primary process:

i40e_dev_alarm_handler(): ICR0: malicious programming detected


  *   Additionally, we saw no example of using external physical shared memory on secondary processes.

DPDK test applications show usage of anonymous private memory on the primary process only.

What are we missing?
Can you please advise?

Thanks,
Asaf

[Radware]
Asaf Sinai
ND SW Engineer
Email: asafsi@radware.com<mailto:asafsi@radware.com>
T:+972-72-3917050
M:+972-50-6518541
F:+972-3-6488662
[Check out the latest and greatest from Radware]<https://www.radware.com/Resources/CampaignRedirector.html>

www.radware.com<https://www.radware.com>

[Blog]<https://blog.radware.com/>  [https://www.radware.com/images/signature/linkedin.jpg] <https://www.linkedin.com/companies/165642> [https://www.radware.com/images/signature/twitter.jpg] <file://twitter.com/radware>   [youtube] <https://www.youtube.com/user/radwareinc>
Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
P Please consider your environmental responsibility before printing this e-mail


[-- Attachment #2: Type: text/html, Size: 23054 bytes --]