[dpdk-dev] Preemption in rings

[dpdk-dev] Preemption in rings

[Carlos Franco]

Hello

I have read the documentation and the answer of Olivier about a question to clarify the preemption restriction in the rings.

He wrote:

- a pthread doing multi-producers enqueues on a given ring must not  be preempted by another pthread doing a multi-producer enqueue on the same ring.

- a pthread doing multi-consumers dequeues on a given ring must not be preempted by another pthread doing a multi-consumer dequeue on  the same ring.

I suppose this is a something to be taken into account with threads not created by the DPDK rte, because the threads of DPDK are attached to a core and will never be scheduled into the cores of other DPDK threads, no?

I mean, lets only take into account the enqueuing. if there is a ring R where ONLY the DPDK threads enqueue packets, even if they are interrupted (IO access, sleep, mutex... ), and other non DPDK threads are scheduled into their cores, as long as only DPDK threads enqueue in the ring it is safe, no?

If not, would prohibit the access to the DPDK used cores (isocpus) work?

Thanks

Carlos
From olivier.matz@6wind.com  Mon Jan 20 14:08:44 2014
Return-Path: <olivier.matz@6wind.com>
Received: from mail.droids-corp.org (zoll.droids-corp.org [94.23.50.67])
 by dpdk.org (Postfix) with ESMTP id 0D76B58D2
 for <dev@dpdk.org>; Mon, 20 Jan 2014 14:08:44 +0100 (CET)
Received: from was59-1-82-226-113-214.fbx.proxad.net ([82.226.113.214]
 helo=[192.168.0.10]) by mail.droids-corp.org with esmtpsa
 (TLS1.0:DHE_RSA_CAMELLIA_256_CBC_SHA1:256) (Exim 4.80)
 (envelope-from <olivier.matz@6wind.com>)
 id 1W5Ecj-0005ha-KV; Mon, 20 Jan 2014 14:10:44 +0100
Message-ID: <52DD20F3.8070801@6wind.com>
Date: Mon, 20 Jan 2014 14:13:23 +0100
From: Olivier MATZ <olivier.matz@6wind.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20131005 Icedove/17.0.9
MIME-Version: 1.0
To: Carlos Franco <kralosf@gmail.com>
References: <2AF1C8858EE34CAF98E426C4B8132229@aervox.com>
In-Reply-To: <2AF1C8858EE34CAF98E426C4B8132229@aervox.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Preemption in rings
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 20 Jan 2014 13:08:44 -0000

Hi Carlos,

On 01/18/2014 02:10 PM, Carlos Franco wrote:
 > - a pthread doing multi-producers enqueues on a given ring must
 >   not be preempted by another pthread doing a multi-producer
 >   enqueue on the same ring.
 >
 > - a pthread doing multi-consumers dequeues on a given ring must
 >   not be preempted by another pthread doing a multi-consumer
 >   dequeue on the same ring.
 >
 > I suppose this is a something to be taken into account with
 > threads not created by the DPDK rte, because the threads of DPDK
 > are attached to a core and will never be scheduled into the cores
 > of other DPDK threads, no?

Yes.

 > I mean, lets only take into account the enqueuing. if there is a
 > ring R where ONLY the DPDK threads enqueue packets, even if they
 > are interrupted (IO access, sleep, mutex... ), and other non DPDK
 > threads are scheduled into their cores, as long as only DPDK
 > threads enqueue in the ring it is safe, no?

It's safe, but it may cause performance issue. For instance, if a
DPDK pthread is interrupted by the kernel during a critical period
of an enqueue, all other DPDK pthreads enqueueing on this ring will
block.

Of course, this could also happen with a rte_spinlock().

 > If not, would prohibit the access to the DPDK used
 > cores (isocpus) work?

Yes, using "isolcpu" is a good idea to avoid the problem described
above.

 From my experience, using a simple "taskset" to force the other
applications to run on different cores than the DPDK is enough
to avoid significant performance issues. Indeed, the only problem
would be related to kernel work (interruptions and kernel threads),
and they shouldn't last long enough to fill the queues.

Regards,
Olivier