From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1BE81430E6; Thu, 24 Aug 2023 05:12:03 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BFC6540EE1; Thu, 24 Aug 2023 05:12:02 +0200 (CEST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2040.outbound.protection.outlook.com [40.107.20.40]) by mails.dpdk.org (Postfix) with ESMTP id D0D5340C35 for ; Thu, 24 Aug 2023 05:12:00 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=A/Emi9hCcpuv03QnVEOm8LBZ6dkr/tanQIAf4O3mGbUwNiwgffZ2A2lIR0b+tv6u7Tsrj/QDNU7NDKilXQfAoKY6lX+XIZESJ7B+dpjVuKTPr4lnXbkW0sp85IwYMvtsBmBbfmFR2L9+7Jy1P6FgemZUG6R99Xk/au5CgOPqoHoIsaKawCt6zehq4jyARYZjDyUjCld2VFeOluKmqN13fQXvG0t/hZG5lX7Fw9p4ohzlyt3Vv+euFCnYlepBTLYCJlD38QFmANIFERfDAFseYqFZ7q/40CPqUreSOkS8ca+vcgdrc5JS9mxMXdwUyVF1NAPdNxyTFUoQIpozhPyA+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l9LtNEDbXdU6TuWQOb4pVPPLMlvc4ZkPcfaOc7eaD/o=; b=d+sLwNxv3tfo5E1eMFVPusIZvgIc9o0TrVgWtwZlX4LHs3pEL67nPs5iqEdEEoOu6GLzTmIr/gZEC/50sHsPdikp/UTiq85oM3kFGtgChHUs0q0HuAQ5XyQ5NlIuJ5mb2WFVzQ8hAcJCQN7ymjEEBcWn/IYeW/GP7b3NlfUdk57RNNaGe1eLAG/4WbsuuXInnnuz0njRsGyV3GzJLmdPwCAefhXQWh3GmHNiVHq3+Q17iUqbTRkVQICm6nG/7AEq7iERZNlGB7Z/QVJr1LizsC0Ix+D519Ls4UlhkkBVFNVFC6buAf26dMONQ17khmxzE082Zlr27gZs/LjTAQHp5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l9LtNEDbXdU6TuWQOb4pVPPLMlvc4ZkPcfaOc7eaD/o=; b=uwqq+rsQnfnz42jpVcsdZJjeOvJp2W9d/0rtiZ2oIxk1F+/hpSm0ulB8pMTy7VzUqvO/q1h7lidt8oEeQPjx3MQKPT8qmfAKpY1ZU+P/2z0KRJpvkzVTmeYtf5yF3x975Okl5TZB3shRly9evUjA5jpEw3rtMD6Pb7CMPndVHNQ= Received: from AS8PR08MB7718.eurprd08.prod.outlook.com (2603:10a6:20b:50a::22) by DB9PR08MB7535.eurprd08.prod.outlook.com (2603:10a6:10:303::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Thu, 24 Aug 2023 03:11:58 +0000 Received: from AS8PR08MB7718.eurprd08.prod.outlook.com ([fe80::70e8:2daa:5a39:dc50]) by AS8PR08MB7718.eurprd08.prod.outlook.com ([fe80::70e8:2daa:5a39:dc50%4]) with mapi id 15.20.6699.027; Thu, 24 Aug 2023 03:11:58 +0000 From: Feifei Wang To: Stephen Hemminger CC: "dev@dpdk.org" , nd , nd Subject: RE: [PATCH v11 0/4] Recycle mbufs from Tx queue into Rx queue Thread-Topic: [PATCH v11 0/4] Recycle mbufs from Tx queue into Rx queue Thread-Index: AQHZ1MobBs/y3i8Lz0+4fawwopu2tK/2WAaAgAJq+WA= Date: Thu, 24 Aug 2023 03:11:58 +0000 Message-ID: References: <20220420081650.2043183-1-feifei.wang2@arm.com> <20230822072710.1945027-1-feifei.wang2@arm.com> <20230822065901.0bb24be6@hermes.local> In-Reply-To: <20230822065901.0bb24be6@hermes.local> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 2F0CF250A474DF498E4EFC3AFFBFFE6F.0 x-checkrecipientchecked: true authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: AS8PR08MB7718:EE_|DB9PR08MB7535:EE_ x-ms-office365-filtering-correlation-id: 9c1d189e-f087-48d4-914e-08dba44fe0d6 nodisclaimer: true x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 15x6d2GejF2SFnvL59fLyZ+JiEoYn3VCC73rz3LIJy1Zcvt7HBxM+9a3cXiR94EE4od/88h9U3PyxAQxNR8ziqI92YmqZ5OK/mPa8HrfKeOAmflZ7QQYzGeqU2WfIKiBXg3a9faHeYuWKTmds2q9W6Ft0+1ZKz/LHhaL1Hx2UqxYPbIH1ZYKI3cHThaPk7Sx98nqo15biq4lmqnaCiD9lBp0TKatQhc7aDGycI3tXMl/yGzUCh5Pe0NnBLTYp7EqFwOAZ1/vh5Hj6Mqdc9DTGZtZiLAXSWUpD996Td8IYJb7HQSjPdxmQmgvPQcvIVm9rX9dR258eNGrXVwzKPfT7mXfj0fdhtyUJJbUDYmuMtiPUD0NiUxnXOhkcL5TZx81VU6/8YzoptoPWNlaybBIS/jaBYKKX8QGP/zqzLUFK939cAdrOy14joQigkMW9/L//gq58e8jAiH/zrtwzmdo9yejBvRqwjfxpOtS0p6WtOcgW97MDZIItewuBEyyKtEUfbK3Golo05HHV+UxkzIJH1eB5IGVd6UOsqTL7PbPT67nG9G0rWmO3ptPAabs6ME6ljt//7TWDLT8NO/u3ws0ZfNMP+wGnj3ww/qlQ7uA9mg= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AS8PR08MB7718.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(136003)(366004)(376002)(346002)(39860400002)(396003)(451199024)(186009)(1800799009)(66446008)(64756008)(54906003)(66556008)(76116006)(66476007)(66946007)(316002)(6916009)(122000001)(478600001)(38100700002)(55016003)(26005)(38070700005)(71200400001)(41300700001)(53546011)(86362001)(7696005)(2906002)(6506007)(9686003)(966005)(4326008)(8676002)(8936002)(52536014)(5660300002)(83380400001)(33656002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?/3vCf9VYmzmkj/2wRc7YKFY63rNI+axKwYVgN+971mr7PR1w87/my1YlS493?= =?us-ascii?Q?g/hdy2/Zr89WCRTSqVBDzZrIHSKYyqEpKDX1hV7WiE5ueM85Ob1EjwKcfDvP?= =?us-ascii?Q?Ki0GBjY+9PFaSXijEjJ4oaXHtmmzxfx5KZNLb3nb84O+OignAb2Lmp4wWeho?= =?us-ascii?Q?f/bA8VHAi+N2S/7eBO/DVUE29jqLM7RcgibFgksZ4y1eqTS3MFVG1GYnNzjK?= =?us-ascii?Q?6jaGFNQSez7o8g0oObqOnv836aIrIhQJ3N5HPfbFPyZ+wUDFDpGAZkFSPQFp?= =?us-ascii?Q?rAtSyObfZ1K+X5XDBQXbv3bZ8Aigs/W74M2pGdHlexMReQP0ymFWWAXiYhQw?= =?us-ascii?Q?B6q27yFrp3AFBysvCteHjOJ8UlLs7oow/5Ep6Qsd3tvNIrlBF9q6LCzjnf0L?= =?us-ascii?Q?YtHgdYwF8M4qveMSaGY3/ovGJhSuLbgjt4LiJVcGJwVtDItRUUmd8jLyiPmR?= =?us-ascii?Q?Pa4yEjryxBUoHXgx6I1ETIl59N2wlfDLInjIE49T0j/t7yGgAEcqqbV9Alav?= =?us-ascii?Q?GEs3KPGDLDvsCqrvnbEhnCAMS5hQEKlXr4PiRcZAmohUqmJwELIfydVSwhq1?= =?us-ascii?Q?kS6oZc9L85O2rljZ8UUuSkcBojCUsHn1Zp0JReaG48hK1N92iEKJaDRvmpQL?= =?us-ascii?Q?Y9Y4k3jxmvx5onwpyXnQUHpRX9jhLFOBsgj23mSdc+Kn6fS+QrBkUL0t2KkY?= =?us-ascii?Q?0kEaEz5CiEvW9S1Sh5EMnIliMP/MOK3Hzg6G6L+Uky1xU0mOzXZvuw4+MWhq?= =?us-ascii?Q?QJtLQXqrKgX9HtB1Wm/v0XiK//W+c2EuTohKvovbDv2Tk7Q3TeV9yOIEI8Zx?= =?us-ascii?Q?XCpTM+BN56MrS3fRhaJFl0WSyn2JS6fxI3uktieIMTRtV/PKd4F2UP3jM16o?= =?us-ascii?Q?A42xgDnk4Yx8iZ09GV1hMStw9oM2OJaaMQhlHUpeQMegEpmVpzKwtN+KydA/?= =?us-ascii?Q?ewzXqS8lH2XFeGdrNX5BYxMb0DkWMemqan7hHotV6AOC7AKoyhavT4f4XMi3?= =?us-ascii?Q?4Si6k7s5bl1SpueCaaHBIuK3MSSlDWQBXSsJgC1l+gzOz+tFYU+uxEWNalPw?= =?us-ascii?Q?9IE4SSjlsMQs/COtq1Hzjt6LP9t19B//sU+zcMqHPjpjLKaS3gC8MPV5xlVt?= =?us-ascii?Q?TmfW6JhvhI+47I6BUxcivzASZE0vBA9y9cI+tKzKhUyFAIm+jZGAO6Cz9Heq?= =?us-ascii?Q?f+kDaRaZvPjWW0GfeyTpV2LSjVVJ/846U8x6EXcl6kzdqk9PG1BeDtsJIHoQ?= =?us-ascii?Q?Bh76iL5NEss8UrN0ce10Er6DkEVAL8983pYeZZnaGym5WCn+c7uC6cKmPAAa?= =?us-ascii?Q?99UduenkjlI9c30SQaZ+4NVWz6p2sL6Mw9Q6NkB4DfdOqKLRCvUQ9KDI2yEJ?= =?us-ascii?Q?2Biru+5nss/sjRTz1VKjdv9DrI32KR9X+vXMrbETymztEBydsPJ+7RpUqYFB?= =?us-ascii?Q?JENqXbDUJy4r7BIQgI25YuuJf+uJxm3LyMk3uUXspbVtI0/SXUMY/2QT2UjP?= =?us-ascii?Q?ofSLCcAfVcjt9Gz/RmC3pBtTafObtJGLIVBTCM1p7RfH1/yxLKBoMKvmsXjG?= =?us-ascii?Q?l/H2NWZd/DGy4tm4PE0Cznej/Nl7H5untuzbRkl+?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR08MB7718.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9c1d189e-f087-48d4-914e-08dba44fe0d6 X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Aug 2023 03:11:58.4141 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: QAlntDzsconMnD9neg9yEcE/gqsKRRIfSnjmaxkBagypPiOUcXDjYNkIbiEhacYpKP1OcYxIRwdik4IS/OFULw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB7535 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Stephen Hemminger > Sent: Tuesday, August 22, 2023 9:59 PM > To: Feifei Wang > Cc: dev@dpdk.org; nd > Subject: Re: [PATCH v11 0/4] Recycle mbufs from Tx queue into Rx queue >=20 > On Tue, 22 Aug 2023 15:27:06 +0800 > Feifei Wang wrote: >=20 > > Currently, the transmit side frees the buffers into the lcore cache > > and the receive side allocates buffers from the lcore cache. The > > transmit side typically frees 32 buffers resulting in 32*8=3D256B of > > stores to lcore cache. The receive side allocates 32 buffers and > > stores them in the receive side software ring, resulting in 32*8=3D256B > > of stores and 256B of load from the lcore cache. > > > > This patch proposes a mechanism to avoid freeing to/allocating from > > the lcore cache. i.e. the receive side will free the buffers from > > transmit side directly into its software ring. This will avoid the > > 256B of loads and stores introduced by the lcore cache. It also frees > > up the cache lines used by the lcore cache. And we can call this mode > > as mbufs recycle mode. >=20 > Isn't the recycle ring just another cache? Why is the lcore cache slower? > Could we fix the general case there? Here lcore cache means the mempool lcore cache for each lcore: mp->local_c= ache[lcore_id]; For each buffer allocate from mempool and free into mempool, the thread wil= l firstly try to do Memory copy from or into lcore cache. We do not say lcore cache is slower, we means do memory copy from or into l= core cache will cost much CPU cycle, and mbuf recycle can bypass these memory copy. For generic case , we try to use zero copy to optimize, but the performance= is worse than mbufs recycle: For general path:=20 Rx: 32 pkts memcpy from mempool cache to rx_sw_ring Tx: 32 pkts memcpy from tx_sw_ring to temporary variable + = 32 pkts memcpy from temporary variable to mempool cache For ZC API used in mempool: Rx: 32 pkts memcpy from mempool cache to rx_sw_ring Tx: 32 pkts memcpy from tx_sw_ring to zero-copy mempool cac= he Refer link: http://patches.dpdk.org/project/dpdk/patch/2023= 0221055205.22984-2-kamalakshitha.aligeri@arm.com/ For mbufs recycle: Rx/Tx: 32 pkts memcpy from tx_sw_ring to rx_sw_ring