From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BD1EEA0573; Thu, 5 Mar 2020 00:21:54 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 16B062C4F; Thu, 5 Mar 2020 00:21:54 +0100 (CET) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00047.outbound.protection.outlook.com [40.107.0.47]) by dpdk.org (Postfix) with ESMTP id 52B013B5 for ; Thu, 5 Mar 2020 00:21:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HZoBrkvf0rTIPI5vW3u0BAJTgxvijWTDssfh848gU28=; b=TW7pVjuFgk3Iq2+69wgc+ZBGTVX7xrqyAtdz0dqbFakPB6yZJTSwfxutCb8AES5DQJRpw0elbLxnXrmH/S8RjXBpsuFykIZRI9y8N8mOHLVDxFC59MSoxSNK2zvNacjQM3FA0JosrstFKf9o4QnmYwkLuYqgf9hyD+7ylIx/VBE= Received: from DB8PR06CA0048.eurprd06.prod.outlook.com (2603:10a6:10:120::22) by HE1PR0802MB2524.eurprd08.prod.outlook.com (2603:10a6:3:e0::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.15; Wed, 4 Mar 2020 23:21:51 +0000 Received: from DB5EUR03FT051.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:120:cafe::66) by DB8PR06CA0048.outlook.office365.com (2603:10a6:10:120::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.15 via Frontend Transport; Wed, 4 Mar 2020 23:21:51 +0000 Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT051.mail.protection.outlook.com (10.152.21.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2793.11 via Frontend Transport; Wed, 4 Mar 2020 23:21:51 +0000 Received: ("Tessian outbound da94dc68d1bb:v42"); Wed, 04 Mar 2020 23:21:51 +0000 X-CR-MTA-TID: 64aa7808 Received: from ea3e044b1e29.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 3BC32413-3322-4C51-ADD5-1EBD7D318CD4.1; Wed, 04 Mar 2020 23:21:46 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id ea3e044b1e29.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 04 Mar 2020 23:21:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DP3hS5kGwSwITrVuSRxvLfldza8nor4gMvqKYwWS+5Wbfkh0HsdJMtJUtg+RAiMkRA6U9z24Lrv1gkBiw3APZZARGFI6m92tNjFAQ4oLmGP2YfsIYruSmKxIX2A9y38r3s5QfYc87sHiRJna/5I4fuHIvK56lQNB4hjLy+gPip3opiljsQwb9vtcURIsICp1HAV7WELqhf5Zj8m3KgQPZ8BQLMV3lP8/BphhSrEcKN4cKhBp7xm7/Nb9lrAu4hg+20RqFUsGbUPFpRzRKesc4I1jeeGAzHvnPt3Xz0fbHTsM1UgoAYlsjFE2EaLH3Im2+97gf/97VUrw7eV6Kuh6jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HZoBrkvf0rTIPI5vW3u0BAJTgxvijWTDssfh848gU28=; b=WHT4Ms54Nmap/x1rP5aAWcblZ9xJ84SSqAD+tC3YlSlGwbF+bRsR13MOZz5uC/01TinDdRPxnO3Jbbh1gwLL6Pi4ZZ343KW+HWi7BgsM2/cfxbUdynHCvJTJYxWsneei2AdldR2aSIMaKmJ8FBiq3I2ioM4nuyHQU027SJfT61xRD1mdHRIQLc3FVtlG/PH8+AqLqfK18nrpzl7OV4cbgF7TIIdrgYsoLG8efEFFssElKQVkHfRIZTrLLQV4IL+TCiezbQyGaZM1CGj128lRY7vIr4QiilF15tmm0R6JIQektQrMFWWKoBt1cqHRR8UDrvc/hJakzgRpS6us1++LTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HZoBrkvf0rTIPI5vW3u0BAJTgxvijWTDssfh848gU28=; b=TW7pVjuFgk3Iq2+69wgc+ZBGTVX7xrqyAtdz0dqbFakPB6yZJTSwfxutCb8AES5DQJRpw0elbLxnXrmH/S8RjXBpsuFykIZRI9y8N8mOHLVDxFC59MSoxSNK2zvNacjQM3FA0JosrstFKf9o4QnmYwkLuYqgf9hyD+7ylIx/VBE= Received: from VE1PR08MB5149.eurprd08.prod.outlook.com (20.179.30.27) by VE1PR08MB4783.eurprd08.prod.outlook.com (10.255.114.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.19; Wed, 4 Mar 2020 23:21:43 +0000 Received: from VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::29eb:a1be:8f8f:fae2]) by VE1PR08MB5149.eurprd08.prod.outlook.com ([fe80::29eb:a1be:8f8f:fae2%7]) with mapi id 15.20.2750.027; Wed, 4 Mar 2020 23:21:43 +0000 From: Honnappa Nagarahalli To: "Ananyev, Konstantin" , "olivier.matz@6wind.com" CC: Gavin Hu , "dev@dpdk.org" , nd , Honnappa Nagarahalli , nd Thread-Topic: [RFC 1/1] lib/ring: add scatter gather and serial dequeue APIs Thread-Index: AQHV7OTEiOImxZdrNE6vzPRTj2NEk6gvbsUwgAY2BYCAA0L00A== Date: Wed, 4 Mar 2020 23:21:43 +0000 Message-ID: References: <20200224203931.21256-1-honnappa.nagarahalli@arm.com> <20200224203931.21256-2-honnappa.nagarahalli@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 1f4974e0-7733-4000-a6a0-1427b57fd73d.0 x-checkrecipientchecked: true Authentication-Results-Original: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: ce3f4bd8-0731-491d-4e35-08d7c092d1ab X-MS-TrafficTypeDiagnostic: VE1PR08MB4783:|VE1PR08MB4783:|HE1PR0802MB2524: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; x-forefront-prvs: 0332AACBC3 X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(10009020)(4636009)(366004)(39860400002)(376002)(396003)(346002)(136003)(189003)(199004)(6506007)(9686003)(7696005)(81166006)(8936002)(4326008)(186003)(26005)(81156014)(71200400001)(8676002)(2906002)(316002)(52536014)(66446008)(55016002)(54906003)(110136005)(33656002)(86362001)(66946007)(76116006)(66476007)(5660300002)(64756008)(478600001)(66556008); DIR:OUT; SFP:1101; SCL:1; SRVR:VE1PR08MB4783; H:VE1PR08MB5149.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 48ay0q98KDGuk4/B2QqX3pQny2kh4X973f/VWZxH13Dp5W5kDEwrx7rZvlKtr37jeMkgnYPAOgXFDumJHqUPGYSNtUEuYYhC0HBkvFThRvFUllTKyuCbOXU0lSQ0tHO8meajgng6OP/9hVAwieFjYHIKkGVuOwbPKMS2ls4XJxs3gWk4akpKFrUJXcqQeB8w67eh3MxpD13CywXi5zhX0CluTFH/xKp1cwGJeUCOMEiaVRP8K4Hh3zuDGvhtaPi/5E5/4k4BYpMOEzkTXheRQfBLuqG/E1wRObakMk7aDWRx3fmQwTuR+A11gS/uoG/Rf8KNT1JjDz2LGExXM94Do/qq8fPVTicqYnP7UI6uC9fi4ZEBAbZ94/e2HEIxPIm6BWwOL0lWCCEEkcYFs+aF8gPy0ozpBZcOjDMzvMX9GS2l825x8GtFa44HWvE7jNyG x-ms-exchange-antispam-messagedata: qj/ohVH3jcyok3Ax4FoS9KlwqwJN4wZo7t6snxkX6imLLs4Ks8gVrzb4bwi8HDT5xz8m/Upi7XMtc+5hDL8POBaJ8XiJNbcT0B1kdqNIfpdJZ9TEb7pgtGok+FgtMTpuGUIUwb4ih9iN9pv6+ZRmRw== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB4783 Original-Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT051.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; IPV:CAL; SCL:-1; CTRY:IE; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(346002)(136003)(376002)(396003)(39860400002)(189003)(199004)(2906002)(54906003)(6506007)(110136005)(316002)(86362001)(356004)(33656002)(52536014)(5660300002)(4326008)(26005)(55016002)(26826003)(336012)(186003)(70206006)(478600001)(9686003)(81156014)(70586007)(8936002)(81166006)(8676002)(7696005); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0802MB2524; H:64aa7808-outbound-1.mta.getcheckrecipient.com; FPR:; SPF:Pass; LANG:en; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; MX:1; A:1; X-MS-Office365-Filtering-Correlation-Id-Prvs: 211e6ac1-acb6-40c8-7b03-08d7c092cd50 X-Forefront-PRVS: 0332AACBC3 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TjEj5RtIFOxk0wYJZTTFCm8vKa4FtWDiyq5sHb0Jj/BCNW+vet0nFa2DVf05qwsmisFU3UC0rmgcvIyUKMWk/t7VZgVCzU/0nDWtpoAQcSTpk3sSS0iyNjlbWBQDrEqMS3OdjSPHY+rbQXN+M1QU44dJePxkAtNXK6DgUc+TNXxiCNA1YQCdmx8JFs4ZQs6yLW/M4Wj769Wd5I+R0Tw2XI1aT7Z3kSM4VpEB4H+kJ+qRUDLeSOwTR7gum7RKmEgqnkxQMjmO7pGre9eIwAbXM7VI/7mfTr+uwVGQLguWNz+AIDBcFf6Vavv969H8hi5oVekgBe3uf+s1YjvJLYm8hr/zs5upI1tFajO2G8M/YvGZvy/Yivbfh6ZRMPUHHUCLFC6zGvvHWkzJ5Y2KUXg+TE8C9GOTHllpT257xUFkr77bvtfvS2JmJ8ESmPglXdbX X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2020 23:21:51.1577 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ce3f4bd8-0731-491d-4e35-08d7c092d1ab X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0802MB2524 Subject: Re: [dpdk-dev] [RFC 1/1] lib/ring: add scatter gather and serial dequeue APIs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" >=20 > > > > +/** > > > > + * @internal Reserve ring elements to enqueue several objects on > > > > +the ring > > > > + * > > > > + * @param r > > > > + * A pointer to the ring structure. > > > > + * @param esize > > > > + * The size of ring element, in bytes. It must be a multiple of = 4. > > > > + * This must be the same value used while creating the ring. > Otherwise > > > > + * the results are undefined. > > > > + * @param n > > > > + * The number of elements to reserve in the ring. > > > > + * @param behavior > > > > + * RTE_RING_QUEUE_FIXED: Reserve a fixed number of elements > from a > > > ring > > > > + * RTE_RING_QUEUE_VARIABLE: Reserve as many elements as > possible > > > from ring > > > > + * @param is_sp > > > > + * Indicates whether to use single producer or multi-producer re= serve > > > > + * @param old_head > > > > + * Producer's head index before reservation. > > > > + * @param new_head > > > > + * Producer's head index after reservation. > > > > + * @param free_space > > > > + * returns the amount of space after the reserve operation has > finished. > > > > + * It is not updated if the number of reserved elements is zero. > > > > + * @param dst1 > > > > + * Pointer to location in the ring to copy the data. > > > > + * @param n1 > > > > + * Number of elements to copy at dst1 > > > > + * @param dst2 > > > > + * In case of ring wrap around, this pointer provides the locati= on to > > > > + * copy the remaining elements. The number of elements to copy a= t > this > > > > + * location is equal to (number of elements reserved - n1) > > > > + * @return > > > > + * Actual number of elements reserved. > > > > + * If behavior =3D=3D RTE_RING_QUEUE_FIXED, this will be 0 or n = only. > > > > + */ > > > > +static __rte_always_inline unsigned int > > > > +__rte_ring_do_enqueue_elem_reserve(struct rte_ring *r, unsigned > > > > +int esize, > > > > > > > > > I do understand the purpose of reserve, then either commit/abort for > > > serial sync mode, but what is the purpose of non-serial version of > reserve/commit? > > In RCU, I have the need for scatter-gather feature. i.e. the data in > > the ring element is coming from multiple sources ('token' is generated > > by the RCU library and the application provides additional data). If I = do not > provide the reserve/commit, I need to introduce an intermediate memcpy to > get these two data contiguously to copy to the ring element. The sequence= is > 'reserve(1), memcpy1, mempcy2, commit(1)'. > > Hence, you do not see the abort API for the enqueue. > > > > > In serial MP/MC case, after _reserve_(n) you always have to do > > > _commit_(n) - you can't reduce number of elements, or do _abort_. > > Agree, the intention here is to provide the scatter/gather feature. > > > > > Again you cannot avoid memcpy(n) here anyhow. > > > So what is the point of these functions for non-serial case? > > It avoids an intermediate memcpy when the data is coming from multiple > sources. >=20 > Ok, I think I understand what was my confusion: Yes, the following understanding is correct. > Your intention: > 1) reserve/commit for both serial and non-serial mode - > to allow user get/set contents of the ring manually and avoid > intermediate load/stores. > 2) abort only for serial mode. >=20 > My intention: > 1) commit/reserve/abort only for serial case > (as that's the only mode where we can commit less > then was reserved or do abort). I do not know if there is a requirement on committing less than reserved. I= think, if the size of commit is not known during reservation, may be the r= eservation can be delayed till it is known. If there is no requirement to commit less than reserved, then I do not see = a need for serial APIs for enqueue operation. > 2) get/set of ring contents are done as part of either > reserve(for dequeue) or commit(for enqueue) API calls > (no scatter-gather ability). >=20 > I still think that this new API you suggest creates too big exposure of r= ing > internals, and makes it less 'safe-to-use': > - it provides direct access to contents of the ring. > - user has to specify head/tail values directly. It is some what complex. But, with the support of user defined element size= , I think it becomes necessary to support scatter gather feature (since it = is not a single pointer that will be stored). >=20 > So in case of some programmatic error in related user code, there are les= s > chances it could be catch-up by API, and we can easily end-up with silent > memory corruption and other nasty things that would be hard to > catch/reproduce. >=20 > That makes me wonder how critical is this scatter-gather ability in terms= of > overall RCU performance? > Is the gain provided really that significant, especially if you'll update= the ring > by one element at a time? For RCU, it is 64b token and the size of the user data. Not sure how much d= ifference it will make. I can drop the scatter gather requirement for now. >=20 > > > > > > > > BTW, I think it would be good to have serial version of _enqueue_ too= . > > If there is a good use case, they should be provided. I did not come ac= ross a > good use case. > > > > > > > > > + unsigned int n, enum rte_ring_queue_behavior behavior, > > > > + unsigned int is_sp, unsigned int *old_head, > > > > + unsigned int *new_head, unsigned int *free_space, > > > > + void **dst1, unsigned int *n1, void **dst2) > > > > > > I do understand the intention to avoid memcpy(), but proposed API > > > seems overcomplicated, error prone, and not very convenient for the u= ser. > > The issue is the need to handle the wrap around in ring storage array. > > i.e. when the space is reserved for more than 1 ring element, the wrap > around might happen. > > > > > I don't think that avoiding memcpy() will save us that many cycles > > > here, so > > This depends on the amount of data being copied. > > > > > probably better to keep API model a bit more regular: > > > > > > n =3D rte_ring_mp_serial_enqueue_bulk_reserve(ring, num, &free_space)= ; ... > > > /* performs actual memcpy(), m<=3Dn */ > > > rte_ring_mp_serial_enqueue_bulk_commit(ring, obj, m); > > These do not take care of the wrap-around case or I am not able to > understand your comment. >=20 > I meant that serial_enqueue_commit() will do both: > actual copy of elements to the ring and tail update (no Scatter-Gather), = see > above. RCU does not require the serial enqueue APIs, do you have any use case?