DPDK patches and discussions
 help / color / mirror / Atom feed
From: Asaf Sinai <AsafSi@Radware.com>
To: "dev@dpdk.org" <dev@dpdk.org>
Subject: DPDK 19.11.3 with multi processes and external physical memory: unable to receive traffic in the secondary processes
Date: Tue, 12 Jul 2022 06:05:33 +0000	[thread overview]
Message-ID: <DB9PR01MB9980511CA0952BD4DF089316CC869@DB9PR01MB9980.eurprd01.prod.exchangelabs.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 5146 bytes --]

Hi,

We run DPDK 19.11.3 with multi processes.
When using external memory for DPDK (instead of huge pages), we are unable to receive traffic in the secondary processes.


  *   We have external physical memory regions (excluded from Linux):

[ULP-NG]# cat /proc/cmdline
console=ttyS0,19200 isolcpus=1-127 smp_affinity=1 root=/dev/ram0 rwmode=r net.ifnames=0 biosdevname=0 systemd.show_status=0
memmap=0x1000000!0x60000000 memmap=0x90000000!0x800000000 memmap=0x90000000!0x1800000000 quiet cloud-init=disabled


  *   We make all DPDK initializations in the primary process, including mapping the mentioned memory regions via "/dev/mem".

After these memory mapping, we register the external memory, as described in http://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#support-for-externally-allocated-memory, and allocate memory pools:



...

/* Map physical to virtual */

int memFd = open("/dev/mem", O_RDWR);

extMemInfo[i].pageSize   = pPagesInfo->maxPageSize;

extMemInfo[i].memRegSize = 0x0000000080000000;

extMemInfo[i].memRegAddr = mmap(NULL, extMemInfo[i].memRegSize, PROT_READ | PROT_WRITE,

                             MAP_SHARED, memFd, memRegPhysAddr[i]);



/* Create heap */

sprintf(extMemInfo[i].heapName, "extMemHeapSocket_%u", i);

rv = rte_malloc_heap_create(extMemInfo[i].heapName);



/* Save heap socket ID */

rv = rte_malloc_heap_get_socket(extMemInfo[i].heapName);

extMemInfo[i].socketId = rv;



/* Add memory region to heap */

rv = rte_malloc_heap_memory_add(extMemInfo[i].heapName, extMemInfo[i].memRegAddr,

                                extMemInfo[i].memRegSize, NULL, 0, extMemInfo[i].pageSize);

...

/* Allocate memory pool */

memPool = rte_mempool_create(poolName, nbufs, DPDK_MBUF_SIZE, poolCacheSize,

                             sizeof(struct rte_pktmbuf_pool_private),

                             rte_pktmbuf_pool_init, NULL,

                             und_pktmbuf_init, NULL, extMemInfo[i].socketId, DPDK_NO_FLAGS);

...



Please note, that during calls to "rte_malloc_heap_memory_add", we see the following warnings:


EAL: WARNING! Base virtual address hint (0x2101005000 != 0x7ffff4627000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes
EAL: WARNING! Base virtual address hint (0x210100b000 != 0x7ffff4619000) not respected!
EAL:    This may cause issues with mapping memory into secondary processes



And after executing "rte_mempool_create", physical addresses of the allocated memory zones are bad:


[Jul 12 12:02:17] [1875] extMemConfig: heapName=extMemHeapSocket_0, socketId=256, memRegAddr=0x7fff58000000, memRegSize=2415919104, pageSize=2097152
[Jul 12 12:02:18] [1875] memZone: name=MP_ndPool_0, socket_id=256, vaddr=0x7fffe7e7bec0-0x7fffe7ffffc0, paddr=0xffffffffffffffff-0x1840ff, len=1589504, hugepage_sz=2MB



  *   In the next step, we spawn the secondary processes from the primary one, using fork().

This way, all DPDK data and memory mappings are the same on all processes. For example:

Mapping in primary process:
cat /proc/1875/maps
...
7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5                        /dev/mem
7fff58000000-7fffe8000000 rw-s 800000000 00:06 5                         /dev/mem
...

Mapping in secondary process:
cat /proc/2676/maps
...
7ffec8000000-7fff58000000 rw-s 1800000000 00:06 5                        /dev/mem
7fff58000000-7fffe8000000 rw-s 800000000 00:06 5                         /dev/mem
...


  *   Unfortunately, when traffic is received by the secondary processes, we see the following printout on the primary process:

i40e_dev_alarm_handler(): ICR0: malicious programming detected


  *   Additionally, we saw no example of using external physical shared memory on secondary processes.

DPDK test applications show usage of anonymous private memory on the primary process only.

What are we missing?
Can you please advise?

Thanks,
Asaf

[Radware]
Asaf Sinai
ND SW Engineer
Email: asafsi@radware.com<mailto:asafsi@radware.com>
T:+972-72-3917050
M:+972-50-6518541
F:+972-3-6488662
[Check out the latest and greatest from Radware]<https://www.radware.com/Resources/CampaignRedirector.html>

www.radware.com<https://www.radware.com>

[Blog]<https://blog.radware.com/>  [https://www.radware.com/images/signature/linkedin.jpg] <https://www.linkedin.com/companies/165642> [https://www.radware.com/images/signature/twitter.jpg] <file://twitter.com/radware>   [youtube] <https://www.youtube.com/user/radwareinc>
Confidentiality note: This message, and any attachments to it, contains privileged/confidential information of RADWARE Ltd./RADWARE Inc. and may not be disclosed, used, copied, or transmitted in any form or by any means without prior written permission from RADWARE. If you are not the intended recipient, delete the message and any attachments from your system without reading or copying it, and kindly notify the sender by e-mail. Thank you.
P Please consider your environmental responsibility before printing this e-mail


[-- Attachment #2: Type: text/html, Size: 23054 bytes --]

             reply	other threads:[~2022-07-12  6:05 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-07-12  6:05 Asaf Sinai [this message]
2022-07-12 13:13 ` Burakov, Anatoly
2022-07-14 10:24   ` Asaf Sinai
2022-07-14 10:41     ` Asaf Sinai
2022-07-15 10:17       ` Burakov, Anatoly
2022-07-18 11:58         ` Asaf Sinai
2022-07-25  9:21           ` Burakov, Anatoly
2022-07-25  9:31             ` Asaf Sinai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DB9PR01MB9980511CA0952BD4DF089316CC869@DB9PR01MB9980.eurprd01.prod.exchangelabs.com \
    --to=asafsi@radware.com \
    --cc=dev@dpdk.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).