[dpdk-users] multiple use rte_pktmbuf_free occurs competition

[dpdk-users] multiple use rte_pktmbuf_free occurs competition

[forsakening]

Hi :
I use the below struct to handle the internet packet based on the dpdk-16.04 interfaces. 
And my hardware: Intel(R) Xeon(R) CPU E5-2618L v3 @ 2.30GHz , which has 32 logic cores .
And my platform: CentOs 7.0.

[stuct removed cuz not allowed pic.]

Structure:
Internet Packet --> NIC --> DPDK-RX --> Dispatch --> Handle Packet Threads (1 to N) --> rte_pktmbuf_free

And now I use about 12 or more handle threads, and these threads are set affinity to my intel logic core, also, in order not to schedule by other thread, i isolate these logic cores out.

Cuz the Packet may not be very standard Ethernet Packet, or My 82599ES NIC cannot RSS the packet well, I use only one Mempool, and setup only one rx que.
To handle these packet, I use one dispatch thread to "Decode" the packet and  dispatch them to my handle threads.
And then , question comes，when these handle threads free the packet, they are “blocked” or "slow" on rte_pktmbuf_free interface.

Below is my "perf result" of one thread:
Samples: 488K of event 'cycles', Event count (approx.): 22076989633                                                                                                                                                                                                                                                        
 78.92%  libvst_driver.so     [.] rte_pktmbuf_free
 .....Other Info

you can see the free interface slow my thread.....

and, annonate of the rte_pktmbuf_free like this:
/*
 * If there are other enqueues in progress that preceded us,
 * we need to wait for them to complete
 */
while (unlikely(r->prod.tail != prod_head)) {
rte_pause();

/* Set RTE_RING_PAUSE_REP_COUNT to avoid spin too long waiting
 * for other thread finish. It gives pre-empted thread a chance
 * to proceed and finish with ring dequeue operation. */
if (RTE_RING_PAUSE_REP_COUNT &&
    ++rep == RTE_RING_PAUSE_REP_COUNT) {
rep = 0;
sched_yield();
}
}

According to the hotspot，the free interface is waiting for other thread to compelete the enque process. 

I know the thread of my  free thread are little more , but can anybody give me some advise about how can i  use the free interface more effectively  ?

Thanks very much !!




Best Regards
forsakening@sina.cn 
MYTtech, NanJing, CHN