From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.lysator.liu.se (mail.lysator.liu.se [130.236.254.3]) by dpdk.org (Postfix) with ESMTP id 6ADE81B5C8 for ; Fri, 30 Nov 2018 21:56:33 +0100 (CET) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id E15EB4003F for ; Fri, 30 Nov 2018 21:56:32 +0100 (CET) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id D09814003B; Fri, 30 Nov 2018 21:56:32 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-0.9 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.1 X-Spam-Score: -0.9 Received: from [192.168.1.59] (host-90-232-161-71.mobileonline.telia.com [90.232.161.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 932B84003A; Fri, 30 Nov 2018 21:56:30 +0100 (CET) To: Honnappa Nagarahalli , Stephen Hemminger Cc: "Van Haaren, Harry" , "dev@dpdk.org" , nd , Dharmik Thakkar , Malvika Gupta , "Gavin Hu (Arm Technology China)" References: <20181122033055.3431-1-honnappa.nagarahalli@arm.com> <20181127142803.423c9b00@xeon-e3> <20181128152351.27fdebe3@xeon-e3> From: =?UTF-8?Q?Mattias_R=c3=b6nnblom?= Message-ID: <5c5db46f-e154-7932-0905-031e153c6016@ericsson.com> Date: Fri, 30 Nov 2018 21:56:30 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Subject: Re: [dpdk-dev] [RFC 0/3] tqs: add thread quiescent state library X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2018 20:56:33 -0000 On 2018-11-30 03:13, Honnappa Nagarahalli wrote: >> >> Reinventing RCU is not helping anyone. > IMO, this depends on what the rte_tqs has to offer and what the requirements are. Before starting this patch, I looked at the liburcu APIs. I have to say, fairly quickly (no offense) I concluded that this does not address DPDK's needs. I took a deeper look at the APIs/code in the past day and I still concluded the same. My partial analysis (analysis of more APIs can be done, I do not have cycles at this point) is as follows: > > The reader threads' information is maintained in a linked list[1]. This linked list is protected by a mutex lock[2]. Any additions/deletions/traversals of this list are blocking and cannot happen in parallel. > > The API, 'synchronize_rcu' [3] (similar functionality to rte_tqs_check call) is a blocking call. There is no option provided to make it non-blocking. The writer spins cycles while waiting for the grace period to get over. > Wouldn't the options be call_rcu, which rarely blocks, or defer_rcu() which never? Why would the average application want to wait for the grace period to be over anyway? > 'synchronize_rcu' also has grace period lock [4]. If I have multiple writers running on data plane threads, I cannot call this API to reclaim the memory in the worker threads as it will block other worker threads. This means, there is an extra thread required (on the control plane?) which does garbage collection and a method to push the pointers from worker threads to the garbage collection thread. This also means the time duration from delete to free increases putting pressure on amount of memory held up. > Since this API cannot be called concurrently by multiple writers, each writer has to wait for other writer's grace period to get over (i.e. multiple writer threads cannot overlap their grace periods). "Real" DPDK applications typically have to interact with the outside world using interfaces beyond DPDK packet I/O, and this is best done via an intermediate "control plane" thread running in the DPDK application. Typically, this thread would also be the RCU writer and "garbage collector", I would say. > > This API also has to traverse the linked list which is not very well suited for calling on data plane. > > I have not gone too much into rcu_thread_offline[5] API. This again needs to be used in worker cores and does not look to be very optimal. > > I have glanced at rcu_quiescent_state [6], it wakes up the thread calling 'synchronize_rcu' which seems good amount of code for the data plane. > Wouldn't the typical DPDK lcore worker call rcu_quiescent_state() after processing a burst of packets? If so, I would more lean toward "negligible overhead", than "a good amount of code". I must admit I didn't look at your library in detail, but I must still ask: if TQS is basically RCU, why isn't it called RCU? And why isn't the API calls named in a similar manner?