From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 95F1642BED; Wed, 31 May 2023 08:10:52 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 2271C40ED7; Wed, 31 May 2023 08:10:52 +0200 (CEST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2089.outbound.protection.outlook.com [40.107.20.89]) by mails.dpdk.org (Postfix) with ESMTP id 5714240A87 for ; Wed, 31 May 2023 08:10:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x51X9HZetX0RQp/FMykwujr42h4kCkdWmHS/H3D8el8=; b=Zj+DzYrfdk0Zr+vg72l6WgGC361BVTkawenrFgE/MxHKnF+CnqVDSzTwPYAYoJ1hLRgA1Vjhn5gbO1zJ5VZ8qBEgyVHUoyji9W7uNy36tNKc9e9k4czFSloMbPggnxVjqXOX7CV5JLdF8/kvEmSkO41GPSpf98Ky1kH1JR/p8g4= Received: from DUZPR01CA0108.eurprd01.prod.exchangelabs.com (2603:10a6:10:4bb::23) by PAXPR08MB7492.eurprd08.prod.outlook.com (2603:10a6:102:2b5::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6433.24; Wed, 31 May 2023 06:10:38 +0000 Received: from DBAEUR03FT049.eop-EUR03.prod.protection.outlook.com (2603:10a6:10:4bb:cafe::3d) by DUZPR01CA0108.outlook.office365.com (2603:10a6:10:4bb::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6455.22 via Frontend Transport; Wed, 31 May 2023 06:10:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DBAEUR03FT049.mail.protection.outlook.com (100.127.142.192) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6455.22 via Frontend Transport; Wed, 31 May 2023 06:10:37 +0000 Received: ("Tessian outbound 5bb4c51d5a1f:v136"); Wed, 31 May 2023 06:10:37 +0000 X-CR-MTA-TID: 64aa7808 Received: from 6ac5bf8c33e1.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id F08B4907-1D8B-4B6F-B6CB-F806C93A80E2.1; Wed, 31 May 2023 06:10:31 +0000 Received: from EUR02-DB5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 6ac5bf8c33e1.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 31 May 2023 06:10:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aXC0KGM+JcVXX+zg23oFgkDdo7UQDuJE7zsgUAcGhhsEmQvAMX+lP2XSLTCPR3OjGOvrUJpF+9vjUpzeVKc4GJnviPHsGmfRtRF16W7fsVFFSPH+8a0J9nHGG91gi4hJ1ugDbqij4fYQn4bwfSN6sMr6zCmnohUf6tkrum8xASEBCmrmCHvNMIdqpgdRkESq11p6uPoVkoVqfARwVQpop6x+4fVbOic+lbk+MKb/b02OYkTrAbCEc4ieh4rslWe0SeTWT8lFJ9ME/lU0F1+f7sG68rvoDwwRrKNIor6msViSUOlDAvP9zKy7Z4awp4vPldAjWM8g2LcxFcI+crf/Sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=x51X9HZetX0RQp/FMykwujr42h4kCkdWmHS/H3D8el8=; b=PbrKRgTkXVfQuFlHpzFXJDTtb3PjNJViR8+GXDH8lVfDgUte/PXvwMen4kkxPMvRh46Utq2FZifj7tCCsNHWvOJuywt05NqA2jVdh0+GwtllQDfT71acSf8bq7Kbk5OSIwDbEVjSal1PDJsgLFOEVLLTuEe3hq+KXciFs8rwjTOEuXd61KFJ8HDCMPPUiEO+x2cTpNK+iBcT1SNXEwcbQ9zpHbWMjBc5GWjD33Qu8CLijnFNM9TqtGGnUNZo4MdisNeSaFggzcprWSwpch+b65n/n51dnaBiHp3njzVvCt42049c3SlGuP0cz0PNCNoN0H6Tw66R2+T+scsRse/Ygg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x51X9HZetX0RQp/FMykwujr42h4kCkdWmHS/H3D8el8=; b=Zj+DzYrfdk0Zr+vg72l6WgGC361BVTkawenrFgE/MxHKnF+CnqVDSzTwPYAYoJ1hLRgA1Vjhn5gbO1zJ5VZ8qBEgyVHUoyji9W7uNy36tNKc9e9k4czFSloMbPggnxVjqXOX7CV5JLdF8/kvEmSkO41GPSpf98Ky1kH1JR/p8g4= Received: from AS8PR08MB7718.eurprd08.prod.outlook.com (2603:10a6:20b:50a::22) by AS8PR08MB8657.eurprd08.prod.outlook.com (2603:10a6:20b:565::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6455.22; Wed, 31 May 2023 06:10:28 +0000 Received: from AS8PR08MB7718.eurprd08.prod.outlook.com ([fe80::7878:ff5f:b10:d4a6]) by AS8PR08MB7718.eurprd08.prod.outlook.com ([fe80::7878:ff5f:b10:d4a6%7]) with mapi id 15.20.6433.022; Wed, 31 May 2023 06:10:28 +0000 From: Feifei Wang To: =?iso-8859-1?Q?Morten_Br=F8rup?= , "thomas@monjalon.net" , Ferruh Yigit , Andrew Rybchenko CC: "dev@dpdk.org" , nd , Honnappa Nagarahalli , Ruifeng Wang , nd Subject: RE: [PATCH v6 1/4] ethdev: add API for mbufs recycle mode Thread-Topic: [PATCH v6 1/4] ethdev: add API for mbufs recycle mode Thread-Index: AQHZju23BTNPFS/xUkaAQdl7HxYGP69rF72AgAjS6JA= Date: Wed, 31 May 2023 06:10:28 +0000 Message-ID: References: <20211224164613.32569-1-feifei.wang2@arm.com> <20230525094541.331338-1-feifei.wang2@arm.com> <20230525094541.331338-2-feifei.wang2@arm.com> <98CBD80474FA8B44BF855DF32C47DC35D8794D@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D8794D@smartserver.smartshare.dk> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: CB788CB9783C4F48B93B7A775D2CFA0F.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: AS8PR08MB7718:EE_|AS8PR08MB8657:EE_|DBAEUR03FT049:EE_|PAXPR08MB7492:EE_ X-MS-Office365-Filtering-Correlation-Id: b59dc7b1-17c9-432d-b020-08db619dc0fb x-ld-processed: f34e5979-57d9-4aaa-ad4d-b122a662184d,ExtAddr x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: MT7NO/72ZzQPya3xgErCBsWJN/raRlYbowPbDO4h3qfiYAtfgUgcJ1guGrw4o7NwyhyP9RihGP/B9ZFlA/h9pONln0lLCDCIQisXYFmPEt58o3LkGcVXeRYVWYp/tr8Y4oe1PfzIRWy6qIFzi2IECzErNhroGjjmgcsBI9oTH9dvUoCtiJ4VDBv8s3KE71/jgVXFQ5IXbQVy0wQpGiEodAdOr+lnrSV9q5nO4ztCtQWDlE5FUdB9+2DUtq1pAnRDR1A9kFJzJ44PWtlRmIi9sbrpmcax2Yb1WftdFcA1gKKbQR1QYkxfrd0YvpX7SC6pTTF3k1q+AKYLHi2Ynz/3KsgiWu/1iSnfuDfYQt3yPs87/ZneIbCDuqKdkgeVdCP1kQNRb8O77lAhnfS/EIVtLttpn2YA3l5QjcHWU16ob/7a148r9E7o9FSWkh7riKo6KFm3khOeKrUMpCzjTxNuLQa5Ix3tnP1c2IGS/DC31MAhkuhNbI4aK9VKNOe+SS6zni0fqKkM/gYMvLsWeCfurPKnDDW9iqlF6l9a4uZDmiSibCvLPz036/dq8ggk1SEvRl4DDyUV838dOZsoIvs/WmPeak7MeuXkv0fP4X9oY4AsQZaSk0aRED9qjfLuwiAi X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR08MB7718.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(376002)(346002)(39860400002)(396003)(366004)(136003)(451199021)(38070700005)(38100700002)(122000001)(33656002)(86362001)(55016003)(7696005)(6506007)(53546011)(26005)(9686003)(186003)(8676002)(8936002)(2906002)(41300700001)(5660300002)(52536014)(110136005)(54906003)(66556008)(71200400001)(4326008)(66476007)(66446008)(64756008)(76116006)(478600001)(316002)(66946007)(66574015)(83380400001); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB8657 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DBAEUR03FT049.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: b775fc3e-e501-4abb-b1f0-08db619dbb6e X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dzaXfoYVVTduYdINXWKRtjnCK+23kTvyfNXh1pz5cFqaCquk+CHmwgva3enHZA/YxH1KrWOwwRFxVOq4kJ8rycN45yumXCsCu7smuQQ2BKGReLpzJcS7+8Qb7sf2BTL+7Z/2ZqwsRnPU6NeBK+ZPWYdWKeoAzCIMpe9zACMCdqq1WjQCan9I+xq5nGYxN388wwVsNQU7iSUWI1I0BCNSe6ctpyN4HMndQF0ASd4IrcXYCTuGkWrfEg/S6qj27vf9xbEzKTNddKyZONi9wTNBJIWa17xk6L/sSBBP3v3/k1tszFaAQzK4p/Jl7DL2zy6dAkLNrgouFIAywZ5sem0r2TY0EFDYjuzPW8n+QwkUwlcRKk0LEuEazVYHCbwCrfPSiy7/LsvGavc2tT7emqeJifGMBPjk82Nx/tcf0174HiQvGoN6ael1bz6IWsJtCvrVGUoLJOzk4cPkHV3r+3wF+nh26VqtZRrMR7ydYJgFuLvZBQqGaa9Pry50hEM2w0UGeUtAxau4hY1ArWoxuMhs/WYQQpRD3CjY64RiOTqC/FlyvfHhHM9GH0gqZarwDveh1WHu7ZFbl7oWPIGgTYUY6L7tDGm1HQO5GvfOeLwrf6FvY1lQOklZyySDN+WjsD7hE+3Y/TpvcOBGTAB3fQrZBB0Lqlws7tCUWPOQj5z5Rf9y5u5rn5NjiDS7B9UaWoKIGY1uffVJy+aCz5mUb29NpFDsSVJ5OfCTM6uuMRqPERESGSJevvm/hTAY+E0OzFqk X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230028)(4636009)(376002)(346002)(39860400002)(396003)(136003)(451199021)(46966006)(36840700001)(40470700004)(70586007)(70206006)(52536014)(316002)(4326008)(41300700001)(110136005)(54906003)(5660300002)(2906002)(8936002)(8676002)(36860700001)(40460700003)(7696005)(478600001)(356005)(40480700001)(55016003)(26005)(6506007)(53546011)(186003)(83380400001)(9686003)(66574015)(336012)(47076005)(33656002)(82740400003)(81166007)(86362001)(82310400005); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 May 2023 06:10:37.8380 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b59dc7b1-17c9-432d-b020-08db619dc0fb X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DBAEUR03FT049.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR08MB7492 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Morten Br=F8rup > Sent: Thursday, May 25, 2023 11:09 PM > To: Feifei Wang ; thomas@monjalon.net; Ferruh > Yigit ; Andrew Rybchenko > > Cc: dev@dpdk.org; nd ; Honnappa Nagarahalli > ; Ruifeng Wang > > Subject: RE: [PATCH v6 1/4] ethdev: add API for mbufs recycle mode >=20 > > From: Feifei Wang [mailto:feifei.wang2@arm.com] > > Sent: Thursday, 25 May 2023 11.46 > > > > Add 'rte_eth_recycle_rx_queue_info_get' and 'rte_eth_recycle_mbufs' > > APIs to recycle used mbufs from a transmit queue of an Ethernet > > device, and move these mbufs into a mbuf ring for a receive queue of > > an Ethernet device. This can bypass mempool 'put/get' operations hence > > saving CPU cycles. > > > > For each recycling mbufs, the rte_eth_recycle_mbufs() function > > performs the following operations: > > - Copy used *rte_mbuf* buffer pointers from Tx mbuf ring into Rx mbuf > > ring. > > - Replenish the Rx descriptors with the recycling *rte_mbuf* mbufs > > freed from the Tx mbuf ring. > > > > Suggested-by: Honnappa Nagarahalli > > Suggested-by: Ruifeng Wang > > Signed-off-by: Feifei Wang > > Reviewed-by: Ruifeng Wang > > Reviewed-by: Honnappa Nagarahalli > > --- >=20 > [...] >=20 > > diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h > > index 2c9d615fb5..c6723d5277 100644 > > --- a/lib/ethdev/ethdev_driver.h > > +++ b/lib/ethdev/ethdev_driver.h > > @@ -59,6 +59,10 @@ struct rte_eth_dev { > > eth_rx_descriptor_status_t rx_descriptor_status; > > /** Check the status of a Tx descriptor */ > > eth_tx_descriptor_status_t tx_descriptor_status; > > + /** Pointer to PMD transmit mbufs reuse function */ > > + eth_recycle_tx_mbufs_reuse_t recycle_tx_mbufs_reuse; > > + /** Pointer to PMD receive descriptors refill function */ > > + eth_recycle_rx_descriptors_refill_t recycle_rx_descriptors_refill; > > > > /** > > * Device data that is shared between primary and secondary > > processes >=20 > The rte_eth_dev struct currently looks like this: >=20 > /** > * @internal > * The generic data structure associated with each Ethernet device. > * > * Pointers to burst-oriented packet receive and transmit functions are > * located at the beginning of the structure, along with the pointer to > * where all the data elements for the particular device are stored in sh= ared > * memory. This split allows the function pointer and driver data to be p= er- > * process, while the actual configuration data for the device is shared. > */ > struct rte_eth_dev { > eth_rx_burst_t rx_pkt_burst; /**< Pointer to PMD receive function */ > eth_tx_burst_t tx_pkt_burst; /**< Pointer to PMD transmit function */ >=20 > /** Pointer to PMD transmit prepare function */ > eth_tx_prep_t tx_pkt_prepare; > /** Get the number of used Rx descriptors */ > eth_rx_queue_count_t rx_queue_count; > /** Check the status of a Rx descriptor */ > eth_rx_descriptor_status_t rx_descriptor_status; > /** Check the status of a Tx descriptor */ > eth_tx_descriptor_status_t tx_descriptor_status; >=20 > /** > * Device data that is shared between primary and secondary > processes > */ > struct rte_eth_dev_data *data; > void *process_private; /**< Pointer to per-process device data */ > const struct eth_dev_ops *dev_ops; /**< Functions exported by PMD > */ > struct rte_device *device; /**< Backing device */ > struct rte_intr_handle *intr_handle; /**< Device interrupt handle */ >=20 > /** User application callbacks for NIC interrupts */ > struct rte_eth_dev_cb_list link_intr_cbs; > /** > * User-supplied functions called from rx_burst to post-process > * received packets before passing them to the user > */ > struct rte_eth_rxtx_callback > *post_rx_burst_cbs[RTE_MAX_QUEUES_PER_PORT]; > /** > * User-supplied functions called from tx_burst to pre-process > * received packets before passing them to the driver for transmission > */ > struct rte_eth_rxtx_callback > *pre_tx_burst_cbs[RTE_MAX_QUEUES_PER_PORT]; >=20 > enum rte_eth_dev_state state; /**< Flag indicating the port state */ > void *security_ctx; /**< Context for security ops */ } > __rte_cache_aligned; >=20 > Inserting the two new function pointers (recycle_tx_mbufs_reuse and > recycle_rx_descriptors_refill) as the 7th and 8th fields will move the 'd= ata' and > 'process_private' pointers out of the first cache line. >=20 > If those data pointers are used in the fast path with the rx_pkt_burst an= d > tx_pkt_burst functions, moving them to a different cache line might have = a > performance impact on those two functions. >=20 > Disclaimer: This is a big "if", and wild speculation from me, because I h= aven't > looked at it in detail! If this structure is not used in the fast path li= ke this, you > can ignore my suggestion below. >=20 > Please consider moving the 'data' and 'process_private' pointers to the > beginning of this structure, so they are kept in the same cache line as t= he > rx_pkt_burst and tx_pkt_burst function pointers. >=20 > I don't know the relative importance of the remaining six fast path funct= ions > (the four existing ones plus the two new ones in this patch), so you coul= d also > rearrange those, so the least important two functions are moved out of th= e > first cache line. It doesn't have to be the two recycle functions that go= into a > different cache line. >=20 > -Morten This is a good question~. By reviewing the code, we find the pointers which= are used for fast path can be mapped to structure 'rte_eth_fp_ops *fpo', this ensures all fast pa= th pointers are in the same Rx/Tx cacheline void eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, const struct rte_eth_dev *dev) { fpo->rx_pkt_burst =3D dev->rx_pkt_burst; fpo->tx_pkt_burst =3D dev->tx_pkt_burst; fpo->tx_pkt_prepare =3D dev->tx_pkt_prepare; fpo->rx_queue_count =3D dev->rx_queue_count; fpo->rx_descriptor_status =3D dev->rx_descriptor_status; fpo->tx_descriptor_status =3D dev->tx_descriptor_status; fpo->recycle_tx_mbufs_reuse =3D dev->recycle_tx_mbufs_reuse; fpo->recycle_rx_descriptors_refill =3D dev->recycle_rx_descriptors_refill; fpo->rxq.data =3D dev->data->rx_queues; fpo->rxq.clbk =3D (void **)(uintptr_t)dev->post_rx_burst_cbs; fpo->txq.data =3D dev->data->tx_queues; fpo->txq.clbk =3D (void **)(uintptr_t)dev->pre_tx_burst_cbs; } Besides rx_queues and tx_queues pointer are important for fast path, other= members of 'data' and 'process_private' are for slow path. So it is not necessary for = these members to be in the cacheline.=20