rte_muf: Inconsistency in the programmer's guide or the code.....?
---------------------------------------------------------
>From the DPDK 1.7.0 programmer's guide we read:
"For a newly allocated mbuf, the area at which the data begins in the
message buffer
is RTE_PKTMBUF_HEADROOM bytes after the beginning of the buffer, which
is cache
aligned."
In the file ./lib/librte_mbuf/rte_mbuf.c and function rte_pktmbuf_init() we
find:
m->pkt.data = (char*) m->buf_addr + RTE_MIN(RTE_PKTMBUF_HEADROOM,
m->buf_len);
Where RTE_PKTMBUF_HEADROOM is configured to be 128 bytes from the file
./config/common_linuxapp:
CONFIG_RTE_PKTMBUF_HEADROOM=128
Question 1:
Does the above invocation of RTE_MIN() cause the programmer's guide
to be inaccurate?
(Saying "in practice it does not matter..." is not an answer)
Question 2:
Why is "packet metadata" implicitly sharing the same bytes of the mbuf
headroom area,
instead of being explicitly kept from being overwritten by a call to
rte_pktmbuf_prepend(pkt, 100);
presuming my metadata is at least 32 bytes long????
The above command would place the packet data area starting at the last
4 bytes of my metadata....
(Saying "you can change CONFIG_RTE_PKTMBUF_HEADROOM at compile time" is
not an answer because it
effects all mbufs not just the one I with my meta-data...)
Question 3:
Why do we write:
#define MBUF_SIZE (2048 + sizeof(struct rte_mbuf) +
RTE_PKTMBUF_HEADROOM)
rte_mempool_create(s, 1024, MBUF_SIZE, 32,0,
rte_pktmbuf_pool_init, NULL,
rte_pktmbuf_init, NULL,
socketid, flags);
instead of:
#define MBUF_SIZE (2048 + sizeof(struct rte_mbuf))
rte_mempool_create(s, 1024, MBUF_SIZE, 32,0,
rte_pktmbuf_pool_init, NULL,
rte_pktmbuf_init, NULL,
socketid, flags);
and have rte_pktmbuf_init() enforce ( + RTE_PKTMBUF_HEADROOM )
automatically?
------------------------------
My reason for asking the above questions stems from my work on a Packet
Framework
Pipeline.
I came to understand that there really isn't an *explicit* declaration
of "allocate this much meta-data space for each packet".
The programmer's guide and mbuf.h describe of headroom and tailroom as a
place to edit
the packet data. An example would be to wrap the packet for tunnelling by
prepending
and appending the data area to be large enought to contain a new header and
checksum.
In fact, while the programmer's guide mentions packet meta-data a few
times, there is no
section which actually describes ****how to make and access**** your very
own packet meta-data.
This addition would be very nice.
The tables in the Packet Framework Pipeline examples all use keys injected
into the meta-data
of the mbuf at RX time to compare a rule against that key (explicitly
stating the "offset
into packet meta-data") and not allowing "offset into packet data".
I actually like this setup because it allows the meta-data key to be
securely analyzed
and copied from the packet data - keeping malformed packets out of the
decision making
process of the tables.
But, in the end, sharing the meta-data area with the packet headroom seems
to be a very
bad idea.
Sincerely,
Daniel Chapiesky
On Thu, Aug 28, 2014 at 08:00:59PM -0400, daniel chapiesky wrote:
> But, in the end, sharing the meta-data area with the packet headroom seems
> to be a very
> bad idea.
>
> Sincerely,
>
> Daniel Chapiesky
You might have picked a good time to inquire about it as some of the Intel
guys are making patches to clean up rte_mbuf during the last couple of weeks
as we speak.
Matthew.
hi, Running primary process, it is successful. $ sudo ./simple_mp -c 1 -n 1 --socket-mem=64 --proc-type=primary EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Setting up memory... EAL: Ask a virtual area of 0x26000000 bytes EAL: Virtual area found at 0x7fffd1200000 (size = 0x26000000) EAL: Ask a virtual area of 0x2200000 bytes EAL: Virtual area found at 0x7fffcee00000 (size = 0x2200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffcea00000 (size = 0x200000) EAL: Ask a virtual area of 0x1a00000 bytes EAL: Virtual area found at 0x7fffcce00000 (size = 0x1a00000) EAL: Ask a virtual area of 0x2200000 bytes EAL: Virtual area found at 0x7fffcaa00000 (size = 0x2200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffca600000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffca200000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9e00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9a00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9600000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9200000 (size = 0x200000) EAL: Requesting 32 pages of size 2MB from socket 0 EAL: TSC frequency is ~2793683 KHz EAL: Master core 0 is ready (tid=f7fe4800) EAL: PCI device 0000:02:01.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: 0000:02:01.0 not managed by UIO driver, skipping EAL: PCI device 0000:02:06.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: 0000:02:06.0 not managed by UIO driver, skipping proc primary APP: Finished Process Init. simple_mp > Running secondary process, it is faied for memzone <RG_MP_log_history> already exists. $ sudo ./simple_mp -c 1 -n 1 --proc-type=secondary --socket-mem=1 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Setting up memory... EAL: Analysing 358 files EAL: Mapped segment 0 of size 0x4000000 EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_log_history> already exists RING: Cannot reserve memory EAL: TSC frequency is ~2793681 KHz EAL: Master core 0 is ready (tid=f7fe4800) EAL: PCI device 0000:02:01.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: Cannot find resource for device EAL: PCI device 0000:02:06.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: Cannot find resource for device proc secandary APP: Finished Process Init. simple_mp >
Hi, 'memzone <RG_MP_log_history> already exists' is not an error for the secondary DPDK process. DPDK EAL initialization creates a mempool for log history. This mempool creation includes creating a ring based on the memzone named RG_MP_log_history. For the secondary DPDK process, if the mempool for log history creation fails because the memzone RG_MP_log_history exits, it looks up the exited mempool and reuse it. Please refer to the implementation of rte_eal_common_log_init() function. From the logs, it seems your secondary DPDK process has initialized successfully. int rte_eal_common_log_init(FILE *default_log) { STAILQ_INIT(&log_history); /* reserve RTE_LOG_HISTORY*2 elements, so we can dump and * keep logging during this time */ log_history_mp = rte_mempool_create(LOG_HISTORY_MP_NAME, RTE_LOG_HISTORY*2, LOG_ELT_SIZE, 0, 0, NULL, NULL, NULL, NULL, SOCKET_ID_ANY, 0); if ((log_history_mp == NULL) && ((log_history_mp = rte_mempool_lookup(LOG_HISTORY_MP_NAME)) == NULL)){ RTE_LOG(ERR, EAL, "%s(): cannot create log_history mempool\n", __func__); return -1; } default_log_stream = default_log; rte_openlog_stream(default_log); return 0; } -----Original Message----- From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of zimeiw Sent: Saturday, August 30, 2014 7:23 AM To: dev@dpdk.org Subject: [dpdk-dev] rte_memzone: memzone_reserve_aligned_thread_unsafe failed when running two process at the same time. hi, Running primary process, it is successful. $ sudo ./simple_mp -c 1 -n 1 --socket-mem=64 --proc-type=primary EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Setting up memory... EAL: Ask a virtual area of 0x26000000 bytes EAL: Virtual area found at 0x7fffd1200000 (size = 0x26000000) EAL: Ask a virtual area of 0x2200000 bytes EAL: Virtual area found at 0x7fffcee00000 (size = 0x2200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffcea00000 (size = 0x200000) EAL: Ask a virtual area of 0x1a00000 bytes EAL: Virtual area found at 0x7fffcce00000 (size = 0x1a00000) EAL: Ask a virtual area of 0x2200000 bytes EAL: Virtual area found at 0x7fffcaa00000 (size = 0x2200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffca600000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffca200000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9e00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9a00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9600000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7fffc9200000 (size = 0x200000) EAL: Requesting 32 pages of size 2MB from socket 0 EAL: TSC frequency is ~2793683 KHz EAL: Master core 0 is ready (tid=f7fe4800) EAL: PCI device 0000:02:01.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: 0000:02:01.0 not managed by UIO driver, skipping EAL: PCI device 0000:02:06.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: 0000:02:06.0 not managed by UIO driver, skipping proc primary APP: Finished Process Init. simple_mp > Running secondary process, it is faied for memzone <RG_MP_log_history> already exists. $ sudo ./simple_mp -c 1 -n 1 --proc-type=secondary --socket-mem=1 EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Support maximum 64 logical core(s) by configuration. EAL: Detected 2 lcore(s) EAL: Setting up memory... EAL: Analysing 358 files EAL: Mapped segment 0 of size 0x4000000 EAL: memzone_reserve_aligned_thread_unsafe(): memzone <RG_MP_log_history> already exists RING: Cannot reserve memory EAL: TSC frequency is ~2793681 KHz EAL: Master core 0 is ready (tid=f7fe4800) EAL: PCI device 0000:02:01.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: Cannot find resource for device EAL: PCI device 0000:02:06.0 on NUMA socket -1 EAL: probe driver: 8086:100f rte_em_pmd EAL: Cannot find resource for device proc secandary APP: Finished Process Init. simple_mp >