Re: [dpdk-dev] [RFC PATCH 0/7] support multi-phtread per lcore

["Jayakumar, Muthurajan" <muthurajan.jayakumar@intel.com>]

Steve, 

Great write up.
Nice explanation of 1) per-lcore numbering and 2) Multi-producer/consumer enqueue -dequeue.

Thanks,

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Cunming Liang
Sent: Wednesday, December 10, 2014 6:05 PM
To: dev@dpdk.org
Subject: [dpdk-dev] [RFC PATCH 0/7] support multi-phtread per lcore


Scope & Usage Scenario
========================  

DPDK usually pin pthread per core to avoid task switch overhead. It gains performance a lot, but it's not efficient in all cases. In some cases, it may too expensive to use the whole core for a lightweight workload. It's a reasonable demand to have multiple threads per core and each threads share CPU in an assigned weight.

In fact, nothing avoid user to create normal pthread and using cgroup to control the CPU share. One of the purpose for the patchset is to clean the gaps of using more DPDK libraries in the normal pthread. In addition, it demonstrates performance gain by proactive 'yield' when doing idle loop in packet IO. It also provides several 'rte_pthread_*' APIs to easy life.


Changes to DPDK libraries
==========================

Some of DPDK libraries must run in DPDK environment.

# rte_mempool

In rte_mempool doc, it mentions a thread not created by EAL must not use mempools. The root cause is it uses a per-lcore cache inside mempool. 
And 'rte_lcore_id()' will not return a correct value.

The patchset changes this a little. The index of mempool cache won't be a lcore_id. Instead of it, using a linear number generated by the allocator.
For those legacy EAL per-lcore thread, it apply for an unique linear id during creation. For those normal pthread expecting to use rte_mempool, it requires to apply for a linear id explicitly. Now the mempool cache looks like a per-thread base. The linear ID actually identify for the linear thread id.

However, there's another problem. The rte_mempool is not preemptable. The problem comes from rte_ring, so talk together in next section.

# rte_ring

rte_ring supports multi-producer enqueue and multi-consumer dequeue. But it's not preemptable. There's conversation talking about this before.
http://dpdk.org/ml/archives/dev/2013-November/000714.html

Let's say there's two pthreads running on the same core doing enqueue on the same rte_ring. If the 1st pthread is preempted by the 2nd pthread while it has already modified the prod.head, the 2nd pthread will spin until the 1st one scheduled agian. It causes time wasting. In addition, if the 2nd pthread has absolutely higer priority, it's more terrible.

But it doesn't means we can't use. Just need to narrow down the situation when it's used by multi-pthread on the same core.
- It CAN be used for any single-producer or single-consumer situation.
- It MAY be used by multi-producer/consumer pthread whose scheduling policy are all SCHED_OTHER(cfs). User SHOULD aware of the performance penalty befor using it.
- It MUST not be used by multi-producer/consumer pthread, while some of their scheduling policies is SCHED_FIFO or SCHED_RR.


Performance
==============

It loses performance by introducing task switching. On packet IO perspective, we can gain some back by improving IO effective rate. When the pthread do idle loop on an empty rx queue, it should proactively yield. We can also slow down rx for a bit while to take more advantage of the bulk receiving in the next loop. In practice, increase the rx ring size also helps to improve the overrall throughput.


Cgroup Control
================

Here's a simple example, there's four pthread doing packet IO on the same core.
We expect the CPU share rate is 1:1:2:4.
> mkdir /sys/fs/cgroup/cpu/dpdk
> mkdir /sys/fs/cgroup/cpu/dpdk/thread0
> mkdir /sys/fs/cgroup/cpu/dpdk/thread1
> mkdir /sys/fs/cgroup/cpu/dpdk/thread2
> mkdir /sys/fs/cgroup/cpu/dpdk/thread3
> cd /sys/fs/cgroup/cpu/dpdk
> echo 256 > thread0/cpu.shares
> echo 256 > thread1/cpu.shares
> echo 512 > thread2/cpu.shares
> echo 1024 > thread3/cpu.shares


-END-

Any comments are welcome.

Thanks

*** BLURB HERE ***

Cunming Liang (7):
  eal: add linear thread id as pthread-local variable
  mempool: use linear-tid as mempool cache index
  ring: use linear-tid as ring debug stats index
  eal: add simple API for multi-pthread
  testpmd: support multi-pthread mode
  sample: add new sample for multi-pthread
  eal: macro for cpuset w/ or w/o CPU_ALLOC

 app/test-pmd/cmdline.c                    |  41 +++++
 app/test-pmd/testpmd.c                    |  84 ++++++++-
 app/test-pmd/testpmd.h                    |   1 +
 config/common_linuxapp                    |   1 +
 examples/multi-pthread/Makefile           |  57 ++++++
 examples/multi-pthread/main.c             | 232 ++++++++++++++++++++++++
 examples/multi-pthread/main.h             |  46 +++++
 lib/librte_eal/common/include/rte_eal.h   |  15 ++
 lib/librte_eal/common/include/rte_lcore.h |  12 ++  lib/librte_eal/linuxapp/eal/eal_thread.c  | 282 +++++++++++++++++++++++++++---
 lib/librte_mempool/rte_mempool.h          |  22 +--
 lib/librte_ring/rte_ring.h                |   6 +-
 12 files changed, 755 insertions(+), 44 deletions(-)  create mode 100644 examples/multi-pthread/Makefile  create mode 100644 examples/multi-pthread/main.c  create mode 100644 examples/multi-pthread/main.h

--
1.8.1.4