From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 355D043C49; Mon, 13 May 2024 18:02:47 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AB62940E09; Mon, 13 May 2024 18:01:47 +0200 (CEST) Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by mails.dpdk.org (Postfix) with ESMTP id E0AF7402CD for ; Mon, 13 May 2024 17:59:57 +0200 (CEST) Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-517ab9a4a13so3440523a12.1 for ; Mon, 13 May 2024 08:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715615997; x=1716220797; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T2339Pngb3r3wo/CbMvWf8rKDOelCDWZBIBkcmNg/WM=; b=Llefj26Pn+/AkQwZwcmWkrvAlzwQdMp1MUBgPDtJnSEziX5oLp+RKbkmD/icvB22Tt X2DYFlteboQ5e2HqdyjZ79spBAxY3SoX+60eq0jt+86orWR8TC5Q/R3mfEzob95uiXoZ ZxaNbL/mfHrdAIjv+NHztN7ujLvmeF9E4MShM/4jQhCiCYCjHqxscXQCWNLJzHHYZcsZ /wdqeDFCSxVKhIzY/WJZe2GhLWnI5KXm/on0B6tu7iz1APlVdZKdA9NMzAKd/Y9ejO4O W4nirPCSo6VJeUp/5n1pcK4tMfWZt1k7Adj3toWpqjLWQ3dgorYN2UWJ5tnaQenNumLh F5Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715615997; x=1716220797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T2339Pngb3r3wo/CbMvWf8rKDOelCDWZBIBkcmNg/WM=; b=f5J3irO7zoTZY/LDjJIK1JwYw7HqZRePYQui8F8G0VYUyqd6DC7LZVyB/cPadACUVR I1aJQvUvQcUYj1Ohp2eX2Fq2eGKZqaLWj50m/rZSkckpBDdssP0MHdtmaZGWJqXQWAz0 SnQmttOsQVNp60utgURIK8DOCwEf7shAPhcNy0Tfh1dO9ohzUuEID+bdDq9Y/ODSD6wL am7WE1WJPj+5DfZJME2WWF/DbphIhw0NSnwS8hjqyxqUOa1yfKx/OmMTFbuMJJ2EjQ8f OPqNauOwk9hyMhC8gmRRIdx/6pODDP9sR8IYHiIwtfaqfTq+IXzTeVMTMXMTknIjmqII ShbQ== X-Gm-Message-State: AOJu0YwAFsXdS2JcF3z8ei7sogNnkNI8Iu//Neqd0a6OAjMOO9ST09fM yyq30eEeuXsbe7dkJ7IzvyH8+T1IHiAlvMPOO8mSODPitkQCNmLyyCQyTQ== X-Google-Smtp-Source: AGHT+IFtQOpXAPTYXYUHWGHTyjJHivw6VspSEvPZNPOysEhKKPJXfOwFA2j79LQJC+mwgBRFb0+SDQ== X-Received: by 2002:a05:6a20:a125:b0:1af:667e:fe3f with SMTP id adf61e73a8af0-1afde09f770mr14028259637.3.1715615996652; Mon, 13 May 2024 08:59:56 -0700 (PDT) Received: from localhost.localdomain (syn-076-032-089-124.res.spectrum.com. [76.32.89.124]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-6f4d2aa16dfsm7715078b3a.93.2024.05.13.08.59.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 May 2024 08:59:56 -0700 (PDT) From: Nandini Persad To: dev@dpdk.org Cc: Honnappa Nagarahalli Subject: [PATCH 9/9] doc: reword rcu library section in prog guide Date: Mon, 13 May 2024 08:59:11 -0700 Message-Id: <20240513155911.31872-10-nandinipersad361@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240513155911.31872-1-nandinipersad361@gmail.com> References: <20240513155911.31872-1-nandinipersad361@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Mon, 13 May 2024 18:01:34 +0200 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org simple syntax changes made to rcu library sectionin programmer's guide Signed-off-by: Nandini Persad --- doc/guides/prog_guide/rcu_lib.rst | 77 ++++++++++++++++--------------- 1 file changed, 40 insertions(+), 37 deletions(-) diff --git a/doc/guides/prog_guide/rcu_lib.rst b/doc/guides/prog_guide/rcu_lib.rst index d0aef3bc16..c7ae349184 100644 --- a/doc/guides/prog_guide/rcu_lib.rst +++ b/doc/guides/prog_guide/rcu_lib.rst @@ -8,17 +8,17 @@ RCU Library Lockless data structures provide scalability and determinism. They enable use cases where locking may not be allowed -(for example real-time applications). +(for example, real-time applications). In the following sections, the term "memory" refers to memory allocated by typical APIs like malloc() or anything that is representative of -memory, for example an index of a free element array. +memory. An example of this is an index of a free element array. Since these data structures are lockless, the writers and readers are accessing the data structures concurrently. Hence, while removing an element from a data structure, the writers cannot return the memory -to the allocator, without knowing that the readers are not -referencing that element/memory anymore. Hence, it is required to +to the allocator without knowing that the readers are not +referencing that element/memory anymore. Therefore, it is required to separate the operation of removing an element into two steps: #. Delete: in this step, the writer removes the reference to the element from @@ -64,19 +64,19 @@ quiescent state. Reader thread 3 was not accessing D1 when the delete operation happened. So, reader thread 3 will not have a reference to the deleted entry. -It can be noted that, the critical sections for D2 is a quiescent state -for D1. i.e. for a given data structure Dx, any point in the thread execution -that does not reference Dx is a quiescent state. +Note that the critical sections for D2 is a quiescent state +for D1 (i.e. for a given data structure Dx, any point in the thread execution +that does not reference Dx is a quiescent state). Since memory is not freed immediately, there might be a need for -provisioning of additional memory, depending on the application requirements. +provisioning additional memory depending on the application requirements. Factors affecting the RCU mechanism ----------------------------------- It is important to make sure that this library keeps the overhead of -identifying the end of grace period and subsequent freeing of memory, -to a minimum. The following paras explain how grace period and critical +identifying the end of grace period and subsequent freeing of memory +to a minimum. The following paragraphs explain how grace period and critical section affect this overhead. The writer has to poll the readers to identify the end of grace period. @@ -119,14 +119,14 @@ How to use this library The application must allocate memory and initialize a QS variable. Applications can call ``rte_rcu_qsbr_get_memsize()`` to calculate the size -of memory to allocate. This API takes a maximum number of reader threads, -using this variable, as a parameter. +of memory to allocate. This API takes a maximum number of reader threads +using this variable as a parameter. Further, the application can initialize a QS variable using the API ``rte_rcu_qsbr_init()``. Each reader thread is assumed to have a unique thread ID. Currently, the -management of the thread ID (for example allocation/free) is left to the +management of the thread ID (for example, allocation/free) is left to the application. The thread ID should be in the range of 0 to maximum number of threads provided while creating the QS variable. The application could also use ``lcore_id`` as the thread ID where applicable. @@ -134,13 +134,13 @@ The application could also use ``lcore_id`` as the thread ID where applicable. The ``rte_rcu_qsbr_thread_register()`` API will register a reader thread to report its quiescent state. This can be called from a reader thread. A control plane thread can also call this on behalf of a reader thread. -The reader thread must call ``rte_rcu_qsbr_thread_online()`` API to start +The reader thread must call the ``rte_rcu_qsbr_thread_online()`` API to start reporting its quiescent state. Some of the use cases might require the reader threads to make blocking API -calls (for example while using eventdev APIs). The writer thread should not +calls (for example, while using eventdev APIs). The writer thread should not wait for such reader threads to enter quiescent state. The reader thread must -call ``rte_rcu_qsbr_thread_offline()`` API, before calling blocking APIs. It +call ``rte_rcu_qsbr_thread_offline()`` API before calling blocking APIs. It can call ``rte_rcu_qsbr_thread_online()`` API once the blocking API call returns. @@ -149,13 +149,13 @@ state by calling the API ``rte_rcu_qsbr_start()``. It is possible for multiple writer threads to query the quiescent state status simultaneously. Hence, ``rte_rcu_qsbr_start()`` returns a token to each caller. -The writer thread must call ``rte_rcu_qsbr_check()`` API with the token to -get the current quiescent state status. Option to block till all the reader +The writer thread must call the ``rte_rcu_qsbr_check()`` API with the token to +get the current quiescent state status. The option to block till all the reader threads enter the quiescent state is provided. If this API indicates that all the reader threads have entered the quiescent state, the application can free the deleted entry. -The APIs ``rte_rcu_qsbr_start()`` and ``rte_rcu_qsbr_check()`` are lock free. +The APIs ``rte_rcu_qsbr_start()`` and ``rte_rcu_qsbr_check()`` are lock-free. Hence, they can be called concurrently from multiple writers even while running as worker threads. @@ -173,7 +173,7 @@ polls till all the readers enter the quiescent state or go offline. This API does not allow the writer to do useful work while waiting and introduces additional memory accesses due to continuous polling. However, the application does not have to store the token or the reference to the deleted resource. The -resource can be freed immediately after ``rte_rcu_qsbr_synchronize()`` API +resource can be freed immediately after the ``rte_rcu_qsbr_synchronize()`` API returns. The reader thread must call ``rte_rcu_qsbr_thread_offline()`` and @@ -181,9 +181,9 @@ The reader thread must call ``rte_rcu_qsbr_thread_offline()`` and quiescent state. The ``rte_rcu_qsbr_check()`` API will not wait for this reader thread to report the quiescent state status anymore. -The reader threads should call ``rte_rcu_qsbr_quiescent()`` API to indicate that +The reader threads should call the ``rte_rcu_qsbr_quiescent()`` API to indicate that they entered a quiescent state. This API checks if a writer has triggered a -quiescent state query and update the state accordingly. +quiescent state query and updates the state accordingly. The ``rte_rcu_qsbr_lock()`` and ``rte_rcu_qsbr_unlock()`` are empty functions. However, these APIs can aid in debugging issues. One can mark the access to @@ -203,40 +203,43 @@ the application. When a writer deletes an entry from a data structure, the write There are several APIs provided to help with this process. The writer can create a FIFO to store the references to deleted resources using ``rte_rcu_qsbr_dq_create()``. The resources can be enqueued to this FIFO using ``rte_rcu_qsbr_dq_enqueue()``. -If the FIFO is full, ``rte_rcu_qsbr_dq_enqueue`` will reclaim the resources before enqueuing. It will also reclaim resources on regular basis to keep the FIFO from growing too large. If the writer runs out of resources, the writer can call ``rte_rcu_qsbr_dq_reclaim`` API to reclaim resources. ``rte_rcu_qsbr_dq_delete`` is provided to reclaim any remaining resources and free the FIFO while shutting down. +If the FIFO is full, ``rte_rcu_qsbr_dq_enqueue`` will reclaim the resources before enqueuing. +It will also reclaim resources on regular basis to keep the FIFO from growing too large. If the writer runs out of resources, the writer can call ``rte_rcu_qsbr_dq_reclaim`` API to reclaim resources. ``rte_rcu_qsbr_dq_delete`` is provided to reclaim any remaining resources and free the FIFO while shutting down. However, if this resource reclamation process were to be integrated in lock-free data structure libraries, it -hides this complexity from the application and makes it easier for the application to adopt lock-free algorithms. The following paragraphs discuss how the reclamation process can be integrated in DPDK libraries. +hides this complexity from the application and makes it easier for the application to adopt lock-free algorithms. + +The following paragraphs discuss how the reclamation process can be integrated in DPDK libraries. In any DPDK application, the resource reclamation process using QSBR can be split into 4 parts: #. Initialization #. Quiescent State Reporting -#. Reclaiming Resources +#. Reclaiming Resources* #. Shutdown The design proposed here assigns different parts of this process to client libraries and applications. The term 'client library' refers to lock-free data structure libraries such at rte_hash, rte_lpm etc. in DPDK or similar libraries outside of DPDK. The term 'application' refers to the packet processing application that makes use of DPDK such as L3 Forwarding example application, OVS, VPP etc.. -The application has to handle 'Initialization' and 'Quiescent State Reporting'. So, +The application must handle 'Initialization' and 'Quiescent State Reporting'. Therefore, the application: -* the application has to create the RCU variable and register the reader threads to report their quiescent state. -* the application has to register the same RCU variable with the client library. -* reader threads in the application have to report the quiescent state. This allows for the application to control the length of the critical section/how frequently the application wants to report the quiescent state. +* Must create the RCU variable and register the reader threads to report their quiescent state. +* Must register the same RCU variable with the client library. +* Note that reader threads in the application have to report the quiescent state. This allows for the application to control the length of the critical section/how frequently the application wants to report the quiescent state. -The client library will handle 'Reclaiming Resources' part of the process. The +The client library will handle the 'Reclaiming Resources' part of the process. The client libraries will make use of the writer thread context to execute the memory -reclamation algorithm. So, +reclamation algorithm. So, the client library should: -* client library should provide an API to register a RCU variable that it will use. It should call ``rte_rcu_qsbr_dq_create()`` to create the FIFO to store the references to deleted entries. -* client library should use ``rte_rcu_qsbr_dq_enqueue`` to enqueue the deleted resources on the FIFO and start the grace period. -* if the library runs out of resources while adding entries, it should call ``rte_rcu_qsbr_dq_reclaim`` to reclaim the resources and try the resource allocation again. +* Provide an API to register a RCU variable that it will use. It should call ``rte_rcu_qsbr_dq_create()`` to create the FIFO to store the references to deleted entries. +* Use ``rte_rcu_qsbr_dq_enqueue`` to enqueue the deleted resources on the FIFO and start the grace period. +* Note that if the library runs out of resources while adding entries, it should call ``rte_rcu_qsbr_dq_reclaim`` to reclaim the resources and try the resource allocation again. The 'Shutdown' process needs to be shared between the application and the -client library. +client library. Note that: -* the application should make sure that the reader threads are not using the shared data structure, unregister the reader threads from the QSBR variable before calling the client library's shutdown function. +* The application should make sure that the reader threads are not using the shared data structure, unregister the reader threads from the QSBR variable before calling the client library's shutdown function. -* client library should call ``rte_rcu_qsbr_dq_delete`` to reclaim any remaining resources and free the FIFO. +* The client library should call ``rte_rcu_qsbr_dq_delete`` to reclaim any remaining resources and free the FIFO. Integrating the resource reclamation with client libraries removes the burden from the application and makes it easy to use lock-free algorithms. -- 2.34.1