[dpdk-dev] Question on DPDK multi-process support

[dpdk-dev] Question on DPDK multi-process support

[Mauro Annarumma]

Hi,
I have some doubts on the DPDK multi-process support:

1) According to the Programmers Guide (pag. 109 - 20.3 Multi-process Limitations), one of the multi-process limitations is:

"All Intel® DPDK processes running as a single application and using shared memory must have distinct coremask arguments. It is not possible to have a primary and secondary instance, or two secondary instances, using any of the same logical cores. Attempting to do so can cause corruption of memory pool caches, among other issues."

Does it mean that I can run a number of DPDK processes at most equal to the number of my CPU logical cores? In the case I run 2 mono-lcore DPDK processes with the same coremask, what are the possible issues?


2) Can a secondary process use the rte_eth_rx_burst over a queue of a NIC port configured in the primary process?

Regards,
 Mauro

 		 	   		  

Re: [dpdk-dev] Question on DPDK multi-process support

[Richardson, Bruce]

> Hi,
> I have some doubts on the DPDK multi-process support:
> 
> 1) According to the Programmers Guide (pag. 109 - 20.3 Multi-process
> Limitations), one of the multi-process limitations is:
> 
> "All Intel(r) DPDK processes running as a single application and using shared
> memory must have distinct coremask arguments. It is not possible to have
> a primary and secondary instance, or two secondary instances, using any of
> the same logical cores. Attempting to do so can cause corruption of
> memory pool caches, among other issues."
> 
> Does it mean that I can run a number of DPDK processes at most equal to
> the number of my CPU logical cores? In the case I run 2 mono-lcore DPDK
> processes with the same coremask, what are the possible issues?
> 
The potential issues are caused by a dependence on the lcore_id internally by Intel DPDK data structures, especially mempools. If two processes use the same lcore they will have the same lcore_id value, and will try and access the same mempool cache which is not thread-safe. This will cause mempool corruption.
NOTE: this applies only to co-operating processes, i.e. those run as primary and secondary processes. There are no mempool issues with running completely independent Intel DPDK processes on the same cores, i.e. processes run using different "--file-prefix=" parameters, since those do not share any memory and data structures. 

> 
> 2) Can a secondary process use the rte_eth_rx_burst over a queue of a NIC
> port configured in the primary process?
> 
Yes. NIC ports and their data structures are shared between co-operating processes. Multiple processes doing RX on the same port is shown in the "examples/multi_process/symmetric_mp" example application.

Re: [dpdk-dev] Question on DPDK multi-process support

[Mauro Annarumma]

Hi Bruce,
   just for dispel
 any doubt: if there is one primary process and N secondary processes, 
and each process uses only one logical core (they have a coremask equal 
to 0x01, 0x02, 0x04, ....), I can run at the same time at most "my CPU logical cores - 1" secondary processes. Is it right? 
Is it true also if I disable the mempool per-core cache?

Thanks for your help, regards.

Mauro

> From: bruce.richardson@intel.com
> To: mauroannarumma@hotmail.it; dev@dpdk.org
> Subject: RE: [dpdk-dev] Question on DPDK multi-process support
> Date: Mon, 17 Feb 2014 16:44:55 +0000
> 
> > Hi,
> > I have some doubts on the DPDK multi-process support:
> > 
> > 1) According to the Programmers Guide (pag. 109 - 20.3 Multi-process
> > Limitations), one of the multi-process limitations is:
> > 
> > "All Intel(r) DPDK processes running as a single application and using shared
> > memory must have distinct coremask arguments. It is not possible to have
> > a primary and secondary instance, or two secondary instances, using any of
> > the same logical cores. Attempting to do so can cause corruption of
> > memory pool caches, among other issues."
> > 
> > Does it mean that I can run a number of DPDK processes at most equal to
> > the number of my CPU logical cores? In the case I run 2 mono-lcore DPDK
> > processes with the same coremask, what are the possible issues?
> > 
> The potential issues are caused by a dependence on the lcore_id internally by Intel DPDK data structures, especially mempools. If two processes use the same lcore they will have the same lcore_id value, and will try and access the same mempool cache which is not thread-safe. This will cause mempool corruption.
> NOTE: this applies only to co-operating processes, i.e. those run as primary and secondary processes. There are no mempool issues with running completely independent Intel DPDK processes on the same cores, i.e. processes run using different "--file-prefix=" parameters, since those do not share any memory and data structures. 
> 
> > 
> > 2) Can a secondary process use the rte_eth_rx_burst over a queue of a NIC
> > port configured in the primary process?
> > 
> Yes. NIC ports and their data structures are shared between co-operating processes. Multiple processes doing RX on the same port is shown in the "examples/multi_process/symmetric_mp" example application.
 		 	   		  

Re: [dpdk-dev] Question on DPDK multi-process support

[Richardson, Bruce]

If you disable mempool caches, you may be able to run more processes than you have cores, but other issues with duplicate core id's might arise, and your application performance will suffer as the memory pools will be very significantly slower.
However, there is no requirement that the lcore_id actually matches the physical core on which the thread is running, so for threads where the core ids will overlap, you could assign a new core id value for some of the them. This can be done in your code by just writing to the global variable _lcore_id, which is defined in eal_thread.c. The core id value must be unique to each thread, and must be between 0 and RTE_MAX_LCORE.
In general, though, it is recommended that for packet processing you run only a single thread pinned to each logical core on a system.

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Mauro
> Annarumma
> Sent: Tuesday, February 18, 2014 12:46 PM
> To: dev@dpdk.org
> Subject: Re: [dpdk-dev] Question on DPDK multi-process support
> 
> Hi Bruce,
>    just for dispel
>  any doubt: if there is one primary process and N secondary processes, and
> each process uses only one logical core (they have a coremask equal to
> 0x01, 0x02, 0x04, ....), I can run at the same time at most "my CPU logical
> cores - 1" secondary processes. Is it right?
> Is it true also if I disable the mempool per-core cache?
> 
> Thanks for your help, regards.
> 
> Mauro
> 
> > From: bruce.richardson@intel.com
> > To: mauroannarumma@hotmail.it; dev@dpdk.org
> > Subject: RE: [dpdk-dev] Question on DPDK multi-process support
> > Date: Mon, 17 Feb 2014 16:44:55 +0000
> >
> > > Hi,
> > > I have some doubts on the DPDK multi-process support:
> > >
> > > 1) According to the Programmers Guide (pag. 109 - 20.3 Multi-process
> > > Limitations), one of the multi-process limitations is:
> > >
> > > "All Intel(r) DPDK processes running as a single application and
> > > using shared memory must have distinct coremask arguments. It is not
> > > possible to have a primary and secondary instance, or two secondary
> > > instances, using any of the same logical cores. Attempting to do so
> > > can cause corruption of memory pool caches, among other issues."
> > >
> > > Does it mean that I can run a number of DPDK processes at most equal
> > > to the number of my CPU logical cores? In the case I run 2
> > > mono-lcore DPDK processes with the same coremask, what are the
> possible issues?
> > >
> > The potential issues are caused by a dependence on the lcore_id internally
> by Intel DPDK data structures, especially mempools. If two processes use
> the same lcore they will have the same lcore_id value, and will try and
> access the same mempool cache which is not thread-safe. This will cause
> mempool corruption.
> > NOTE: this applies only to co-operating processes, i.e. those run as
> primary and secondary processes. There are no mempool issues with
> running completely independent Intel DPDK processes on the same cores,
> i.e. processes run using different "--file-prefix=" parameters, since those do
> not share any memory and data structures.
> >
> > >
> > > 2) Can a secondary process use the rte_eth_rx_burst over a queue of
> > > a NIC port configured in the primary process?
> > >
> > Yes. NIC ports and their data structures are shared between co-operating
> processes. Multiple processes doing RX on the same port is shown in the
> "examples/multi_process/symmetric_mp" example application.
> 

[dpdk-dev] Receive queue full

[François-Frédéric Ozog]

Hi,

I am bumping into a similar problem than the one explained here
(https://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg07684.htm
l): At some point in time, a receive queue gets "FULL" i.e. tail==head
(reading the NIC registers) and the thread associated to that queue cannot
retrieve any packet from it.

The test program derived from L2FWD echoes packets received on one port back
to the same port, it reads up to 32 packets and send them back. The echo
works nicely for a few seconds then queues gets full and stall.

I have found that setting rx_conf.rx_free_thresh down from 32 to 28 (the
multiple of 4 just before 32) avoids the problem and can handle close to
10Mpps per port. 

Test context:
	- 4 socket Xeon E7 4800v2 with 256GB RAM
	- 32GB hugepages reserved, 104 lcores reserved
	- DPDK 1.5.0, testing with the latest from git presents a
performance glitch I can't pinpoint at present.
	- two ports (either 82599ES or X540) loaded at 10Mpps
	- various tests with 2 to 15 receive queues per port
	- various tests with different combinations of RX_PTHRESH,
RX_HTHRESH, RX_WTHRESH

It really looks like a race condition (32 reads, 32 refresh cycle) but I
can't figure it out if it exists.

I'd be glad to get any comment or question on the issue.

Cordially,

François-Frédéric