From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2270CA3160 for ; Fri, 11 Oct 2019 07:04:17 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7B1181E8FE; Fri, 11 Oct 2019 07:04:16 +0200 (CEST) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20084.outbound.protection.outlook.com [40.107.2.84]) by dpdk.org (Postfix) with ESMTP id DA9E51E8F5 for ; Fri, 11 Oct 2019 07:04:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G4HSSZbleI6Cks5kehsg2HyAiZa6O65VEBJDfcy++HI=; b=LeGvERWzjvkbX4qV+cqERfJWX7hRJnzASADh4kAVNncKgSPc+225ko1ZKnopmdM/78kCYunN95/iRmdBRYX0WKZgh2zh+/rz9/TsfqrPy8v96F+8H2raAgrIcG7DdJW+IQliVSOGljg5TN4IIhB+iLhS2GgmI1MTrPltqgXrIlE= Received: from AM6PR08CA0034.eurprd08.prod.outlook.com (2603:10a6:20b:c0::22) by AM0PR08MB5282.eurprd08.prod.outlook.com (2603:10a6:208:129::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Fri, 11 Oct 2019 05:04:12 +0000 Received: from DB5EUR03FT031.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::200) by AM6PR08CA0034.outlook.office365.com (2603:10a6:20b:c0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2347.16 via Frontend Transport; Fri, 11 Oct 2019 05:04:11 +0000 Authentication-Results: spf=temperror (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=none action=none header.from=arm.com; Received-SPF: TempError (protection.outlook.com: error in processing during lookup of arm.com: DNS Timeout) Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT031.mail.protection.outlook.com (10.152.20.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2305.15 via Frontend Transport; Fri, 11 Oct 2019 05:04:10 +0000 Received: ("Tessian outbound 6481c7fa5a3c:v33"); Fri, 11 Oct 2019 05:04:06 +0000 X-CR-MTA-TID: 64aa7808 Received: from dd3f2de8c05e.2 (ip-172-16-0-2.eu-west-1.compute.internal [104.47.13.51]) by 64aa7808-outbound-1.mta.getcheckrecipient.com id 482FD3FA-4B12-44BE-8FD0-D3BF410B17D3.1; Fri, 11 Oct 2019 05:04:01 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-he1eur04lp2051.outbound.protection.outlook.com [104.47.13.51]) by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id dd3f2de8c05e.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Fri, 11 Oct 2019 05:04:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Uz29j7GMS7DUP/G5quCPr9PWq+kosytUXXpoFgQZILVfzj/9G8ovZMVa1sv9cM3HCDYvUyXvSWJucA0T7zZordVqPYMaxRaS6WzmIe+v5qRkzzqwfEWXOxw/g6Kr4qo8GkhuUobtbEPxPW2YOb+5JWvJuGIZJ7RbueNXYtE04ZMXcjpcA5+gBBtILhZVc9maRri8lEO2nx9MAQCKYaICf02ChUTSboOsITVlgjomX8oHIu6ej2r7+IzMlX4Beorz3j2OkPDE3umfvA3MApsWElPKTOMmJaNU+1OKfzz66zNDS09yUf8CBUPlSc1ozPtZHpXlJ35VCF4Ad3NEgN/qLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G4HSSZbleI6Cks5kehsg2HyAiZa6O65VEBJDfcy++HI=; b=cmFfu7ALIYqkKIcGK4cdjjoAZg4XdnmFPAE8Anml8yJ+qVClcoKCMZ7PnSKK/5l7p21MmH1eWFVf4wjGASW9IN0R3LabTNDkl92maIEzE7k8zZpd48t3cmgnU7mk3ipxZZZJ5jEXeaNY/zsFDeSDSuvnkkRQARLCluW+Hc69le1WVak0JcIUznP3GQhJAbfz9PsDOptcrBHKLh2ZbBhTaVQQ4KYEOd1Y4C/ZD1XsSu6OWOZFmDOZQaCgcLoSUSR/3VsFWeiQmQdUn3f2rzKlhUeLrGrePis4JfKUhXzHy1dER7T/hFgdKDlSyWn9eujwavSbVS6qYbyqGmQMU4IiSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G4HSSZbleI6Cks5kehsg2HyAiZa6O65VEBJDfcy++HI=; b=LeGvERWzjvkbX4qV+cqERfJWX7hRJnzASADh4kAVNncKgSPc+225ko1ZKnopmdM/78kCYunN95/iRmdBRYX0WKZgh2zh+/rz9/TsfqrPy8v96F+8H2raAgrIcG7DdJW+IQliVSOGljg5TN4IIhB+iLhS2GgmI1MTrPltqgXrIlE= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.27) by VE1PR08MB5007.eurprd08.prod.outlook.com (20.179.31.96) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2347.16; Fri, 11 Oct 2019 05:03:59 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::8c82:8d9c:c78d:22a6]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::8c82:8d9c:c78d:22a6%7]) with mapi id 15.20.2347.016; Fri, 11 Oct 2019 05:03:59 +0000 From: Honnappa Nagarahalli To: "Ananyev, Konstantin" , "stephen@networkplumber.org" , "paulmck@linux.ibm.com" CC: "Wang, Yipeng1" , "Medvedkin, Vladimir" , "Ruifeng Wang (Arm Technology China)" , Dharmik Thakkar , "dev@dpdk.org" , nd , nd , nd Thread-Topic: [PATCH v3 1/3] lib/ring: add peek API Thread-Index: AQHVeVEbQBVWnKnBNky5vPA5w4G8vadIeNrQgAZwDgCAAs5McIACT7WAgADnlFA= Date: Fri, 11 Oct 2019 05:03:59 +0000 Message-ID: References: <20190906094534.36060-1-ruifeng.wang@arm.com> <20191001062917.35578-1-honnappa.nagarahalli@arm.com> <20191001062917.35578-2-honnappa.nagarahalli@arm.com> <2601191342CEEE43887BDE71AB9772580191970014@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772580191971EBE@irsmsx105.ger.corp.intel.com> <2601191342CEEE43887BDE71AB9772580191975145@irsmsx105.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB9772580191975145@irsmsx105.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: fcc59889-7963-46b3-b0c8-5e28cacabaf1.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 2d44ebd3-c928-4ff4-a58e-08d74e0873bf X-MS-Office365-Filtering-HT: Tenant X-MS-TrafficTypeDiagnostic: VE1PR08MB5007:|VE1PR08MB5007:|AM0PR08MB5282: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; x-forefront-prvs: 0187F3EA14 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(136003)(346002)(396003)(366004)(376002)(199004)(189003)(8936002)(25786009)(476003)(8676002)(81156014)(14444005)(74316002)(305945005)(7736002)(256004)(229853002)(14454004)(81166006)(9686003)(486006)(66446008)(86362001)(6436002)(66066001)(2201001)(71200400001)(71190400001)(66476007)(66556008)(64756008)(66946007)(11346002)(446003)(55016002)(76116006)(478600001)(186003)(5660300002)(316002)(26005)(3846002)(2501003)(33656002)(4326008)(6116002)(6246003)(52536014)(110136005)(76176011)(99286004)(7696005)(2906002)(6506007)(102836004)(54906003); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB5007; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: /66bx5MqBDvWYJ1nhgGExKcNSHjuTVgbZz8Txo2Z8yG+MDP9y5R1paTFdaI4+OoxXvO1LO+5+WmDrsNxyv+ek5jXYbZG4/HrppTTuVP4XNY03wXgCmTn3FI4ZvRlpYwwlDZsqWNhX+Fjtkz9/DIQ4+UBSe7GfzBc/RvNnP7PIZvz9y4Oh6zi+Ot5aGHhsTIdkgX46Yb+h0tfjiDX5mEzJbGzOOgMuTkGT11Yv43fzA6uSsCzlRoZZJtMUwNK26+DBnI+EJbd00sGQ7PA2/3x4l6mZLJU7BqD9YJbPQfBHoKtatyxddVlZeEfUlgdK7iGrVLkml6BKe3+hK0z1/HJerym6Fpr3DUEItKe3qnb44pptuS+DDeZJsBls6CXZ0I2NAadzLeoN3KakzNXWBRyAzXUHgkPTde+fyEkw21Q1C8= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB5007 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT031.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(136003)(376002)(396003)(39860400002)(346002)(199004)(189003)(81156014)(14454004)(81166006)(22756006)(4326008)(9686003)(102836004)(97756001)(229853002)(305945005)(7736002)(74316002)(8676002)(2906002)(50466002)(6246003)(186003)(55016002)(478600001)(26826003)(14444005)(86362001)(6506007)(33656002)(25786009)(70206006)(70586007)(76130400001)(126002)(99286004)(2201001)(476003)(66066001)(11346002)(52536014)(47776003)(6116002)(26005)(3846002)(8746002)(110136005)(54906003)(336012)(486006)(446003)(2501003)(5660300002)(63350400001)(23726003)(8936002)(7696005)(76176011)(356004)(46406003)(316002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR08MB5282; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:TempError; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; A:1; MX:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 2873b7a9-5fb0-4666-e475-08d74e086d0e NoDisclaimer: True X-Forefront-PRVS: 0187F3EA14 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +IqZq+Pj4dfVvXabb8/YIcXQWFLbKLXFeSZi/jsjMrTxkaw4CuF3SXoO+pC9v1xYPew4paNvMHZAK65YbClFH6KbBZ6kqjdXVfDhg4Uwe2hSy5enYdDunDv0bhl9uv/o+RsP2hxAeQvpWYKkgH+3sIwrCNo8u35vAJV43SpskA9EHAnM94zNIJQRlc6PzVTKSahYmj3a0SLZHQXFAeStR3cMNCcUvGdHsVC1tel9MajK1HUVDByjSTKT9Du4dU/rJyjBzMtCNgzxAi5n9pNRqGDKORegjGboJC74qpqRx4/2L6taKLVeYlNBo4CzRh92/njzRj/rinY926wZPKo7u3spItDzEOxoOZefBoXFifCxuUQq3H+phvPv6czZtkGY7QAV+2B2WWI1frz8RpXxK56dT3B2gfr34c/B0i5JH24= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Oct 2019 05:04:10.4348 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2d44ebd3-c928-4ff4-a58e-08d74e0873bf X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB5282 Subject: Re: [dpdk-dev] [PATCH v3 1/3] lib/ring: add peek API X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > > > > > > > > > > > > > > > > > Subject: [PATCH v3 1/3] lib/ring: add peek API > > > > > > > > > > > > From: Ruifeng Wang > > > > > > > > > > > > The peek API allows fetching the next available object in the > > > > > > ring without dequeuing it. This helps in scenarios where > > > > > > dequeuing of objects depend on their value. > > > > > > > > > > > > Signed-off-by: Dharmik Thakkar > > > > > > Signed-off-by: Ruifeng Wang > > > > > > Reviewed-by: Honnappa Nagarahalli > > > > > > > > > > > > Reviewed-by: Gavin Hu > > > > > > --- > > > > > > lib/librte_ring/rte_ring.h | 30 > > > > > > ++++++++++++++++++++++++++++++ > > > > > > 1 file changed, 30 insertions(+) > > > > > > > > > > > > diff --git a/lib/librte_ring/rte_ring.h > > > > > > b/lib/librte_ring/rte_ring.h index 2a9f768a1..d3d0d5e18 100644 > > > > > > --- a/lib/librte_ring/rte_ring.h > > > > > > +++ b/lib/librte_ring/rte_ring.h > > > > > > @@ -953,6 +953,36 @@ rte_ring_dequeue_burst(struct rte_ring > > > > > > *r, void > > > > > **obj_table, > > > > > > r->cons.single, available); } > > > > > > > > > > > > +/** > > > > > > + * Peek one object from a ring. > > > > > > + * > > > > > > + * The peek API allows fetching the next available object in > > > > > > +the ring > > > > > > + * without dequeuing it. This API is not multi-thread safe > > > > > > +with respect > > > > > > + * to other consumer threads. > > > > > > + * > > > > > > + * @param r > > > > > > + * A pointer to the ring structure. > > > > > > + * @param obj_p > > > > > > + * A pointer to a void * pointer (object) that will be fille= d. > > > > > > + * @return > > > > > > + * - 0: Success, object available > > > > > > + * - -ENOENT: Not enough entries in the ring. > > > > > > + */ > > > > > > +__rte_experimental > > > > > > +static __rte_always_inline int rte_ring_peek(struct rte_ring > > > > > > +*r, void **obj_p) > > > > > > > > > > As it is not MT safe, then I think we need _sc_ in the name, to > > > > > follow other rte_ring functions naming conventions > > > > > (rte_ring_sc_peek() or so). > > > > Agree > > > > > > > > > > > > > > As a better alternative what do you think about introducing a > > > > > serialized versions of DPDK rte_ring dequeue functions? > > > > > Something like that: > > > > > > > > > > /* same as original ring dequeue, but: > > > > > * 1) move cons.head only if cons.head =3D=3D const.tail > > > > > * 2) don't update cons.tail > > > > > */ > > > > > unsigned int > > > > > rte_ring_serial_dequeue_bulk(struct rte_ring *r, void > > > > > **obj_table, unsigned int n, > > > > > unsigned int *available); > > > > > > > > > > /* sets both cons.head and cons.tail to cons.head + num */ void > > > > > rte_ring_serial_dequeue_finish(struct rte_ring *r, uint32_t > > > > > num); > > > > > > > > > > /* resets cons.head to const.tail value */ void > > > > > rte_ring_serial_dequeue_abort(struct rte_ring *r); > > > > > > > > > > Then your dq_reclaim cycle function will look like that: > > > > > > > > > > const uint32_t nb_elt =3D dq->elt_size/8 + 1; uint32_t avl, n; > > > > > uintptr_t elt[nb_elt]; ... > > > > > > > > > > do { > > > > > > > > > > /* read next elem from the queue */ > > > > > n =3D rte_ring_serial_dequeue_bulk(dq->r, elt, nb_elt, &avl); > > > > > if (n =3D=3D 0) > > > > > break; > > > > > > > > > > /* wrong period, keep elem in the queue */ if > > > > > (rte_rcu_qsbr_check(dr->v, > > > > > elt[0]) !=3D 1) { > > > > > rte_ring_serial_dequeue_abort(dq->r); > > > > > break; > > > > > } > > > > > > > > > > /* can reclaim, remove elem from the queue */ > > > > > rte_ring_serial_dequeue_finish(dr->q, nb_elt); > > > > > > > > > > /*call reclaim function */ > > > > > dr->f(dr->p, elt); > > > > > > > > > > } while (avl >=3D nb_elt); > > > > > > > > > > That way, I think even rte_rcu_qsbr_dq_reclaim() can be MT safe. > > > > > As long as actual reclamation callback itself is MT safe of cours= e. > > > > > > > > I think it is a great idea. The other writers would still be > > > > polling for the current writer to update the tail or update the > > > > head. This makes it a > > > blocking solution. > > > > > > Yep, it is a blocking one. > > > > > > > We can make the other threads not poll i.e. they will quit > > > > reclaiming if they > > > see that other writers are dequeuing from the queue. > > > > > > Actually didn't think about that possibility, but yes should be > > > possible to have _try_ semantics too. > > > > > > >The other way is to use per thread queues. > > > > > > > > The other requirement I see is to support unbounded-size data > > > > structures where in the data structures do not have a > > > > pre-determined number of entries. Also, currently the defer queue > > > > size is equal to the total > > > number of entries in a given data structure. There are plans to > > > support dynamically resizable defer queue. This means, memory > > > allocation which will affect the lock-free-ness of the solution. > > > > > > > > So, IMO: > > > > 1) The API should provide the capability to support different > > > > algorithms - > > > may be through some flags? > > > > 2) The requirements for the ring are pretty unique to the problem > > > > we have here (for ex: move the cons-head only if cons-tail is also > > > > the same, skip > > > polling). So, we should probably implement a ring with-in the RCU lib= rary? > > > > > > Personally, I think such serialization ring API would be useful for > > > other cases too. > > > There are few cases when user need to read contents of the queue > > > without removing elements from it. > > > Let say we do use similar approach inside TLDK to implement TCP > > > transmit queue. > > > If such API would exist in DPDK we can just use it straightway, > > > without maintaining a separate one. > > ok > > > > > > > > > > > > > From the timeline perspective, adding all these capabilities would > > > > be difficult to get done with in 19.11 timeline. What I have here > > > > satisfies my current needs. I suggest that we make provisions in > > > > APIs now to > > > support all these features, but do the implementation in the coming > releases. > > > Does this sound ok for you? > > > > > > Not sure I understand your suggestion here... > > > Could you explain it a bit more - how new API will look like and > > > what would be left for the future. > > For this patch, I suggest we do not add any more complexity. If > > someone wants a lock-free/block-free mechanism, it is available by crea= ting > per thread defer queues. > > > > We push the following to the future: > > 1) Dynamically size adjustable defer queue. IMO, with this, the > > lock-free/block-free reclamation will not be available (memory allocati= on > requires locking). The memory for the defer queue will be allocated/freed= in > chunks of 'size' elements as the queue grows/shrinks. >=20 > That one is fine by me. > In fact I don't know would be there a real use-case for dynamic defer que= ue > for rcu var... > But I suppose that's subject for another discussion. Currently, the defer queue size is equal to the number of resources in the = data structure. This is unnecessary as the reclamation is done regularly. If a smaller queue size is used, the queue might get full (even after recla= mation), in which case, the queue size should be increased. >=20 > > > > 2) Constant size defer queue with lock-free and block-free reclamation > > (single option). The defer queue will be of fixed length 'size'. If > > the queue gets full an error is returned. The user could provide a 'siz= e' equal > to the number of elements in a data structure to ensure queue never gets = full. >=20 > Ok so for 19.11 what enqueue/dequeue model do you plan to support? > - MP/MC > - MP/SC > - SP/SC Just SP/SC > - non MT at all (only same single thread can do enqueue and dequeue) If MT safe is required, one should use 1 defer queue per thread for now. >=20 > And related question: > What additional rte_ring API you plan to introduce in that case? > - None > - rte_ring_sc_peek() rte_ring_peek will be changed to rte_ring_sc_peek > - rte_ring_serial_dequeue() >=20 > > > > I would add a 'flags' field in rte_rcu_qsbr_dq_parameters and provide > > 2 #defines, one for dynamically variable size defer queue and the other= for > constant size defer queue. > > > > However, IMO, using per thread defer queue is a much simpler way to > achieve 2. It does not add any significant burden to the user either. > > > > > > > > > > > > > > > > > > > > +{ > > > > > > + uint32_t prod_tail =3D r->prod.tail; > > > > > > + uint32_t cons_head =3D r->cons.head; > > > > > > + uint32_t count =3D (prod_tail - cons_head) & r->mask; > > > > > > + unsigned int n =3D 1; > > > > > > + if (count) { > > > > > > + DEQUEUE_PTRS(r, &r[1], cons_head, obj_p, n, void *); > > > > > > + return 0; > > > > > > + } > > > > > > + return -ENOENT; > > > > > > +} > > > > > > + > > > > > > #ifdef __cplusplus > > > > > > } > > > > > > #endif > > > > > > -- > > > > > > 2.17.1