From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B1A9FA0350; Thu, 27 Jan 2022 05:07:03 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 4B11640685; Thu, 27 Jan 2022 05:07:03 +0100 (CET) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2078.outbound.protection.outlook.com [40.107.20.78]) by mails.dpdk.org (Postfix) with ESMTP id 75ED54067C for ; Thu, 27 Jan 2022 05:07:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nPRpfltP066J3isZV5OX4bHX/ZpjvXKITGjOzNWD6LM=; b=Zzq2Com7wPZ2tRkCgEzrx3TUbH1GsboTTG48YrZsiIEXYe1JhlKjd+3fx1OUprlmcbJdBk5+IKxftXWnVCUr52tA0UWpCuUeGZ+z8TKlSMko7/gKdNiMTIjKL5w9k9byh19PiGNaOTyCdxHppLKoUWQh/nS54gVV03N5H1VI260= Received: from AS8PR04CA0192.eurprd04.prod.outlook.com (2603:10a6:20b:2f3::17) by HE1PR0802MB2588.eurprd08.prod.outlook.com (2603:10a6:3:e2::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4909.11; Thu, 27 Jan 2022 04:06:59 +0000 Received: from VE1EUR03FT064.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:2f3:cafe::e8) by AS8PR04CA0192.outlook.office365.com (2603:10a6:20b:2f3::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.15 via Frontend Transport; Thu, 27 Jan 2022 04:06:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT064.mail.protection.outlook.com (10.152.19.210) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.15 via Frontend Transport; Thu, 27 Jan 2022 04:06:58 +0000 Received: ("Tessian outbound 63bb5eb69ee8:v113"); Thu, 27 Jan 2022 04:06:58 +0000 X-CR-MTA-TID: 64aa7808 Received: from 15d522c165b4.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 28F7BC87-218A-4262-B77C-D58EBEA4EAA7.1; Thu, 27 Jan 2022 04:06:52 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 15d522c165b4.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 27 Jan 2022 04:06:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S7g7Glq5/1ofS/TS5vXufq2WQ7Xcnp+6AiNRojMN2DV+Y2u1PPw+Q40PrrvoZyCCsrQZXv6c2zALeipMr14PrhZR6qnT7opT+HhW9f6mdelJuQwfWdWGfABvyyEtRvzHMEcNO1TPFlX7/Lg08NWERkYTT3p13ba4rfG06yD+HVMcmtAT/JfSIgt0GBF7ttbY0H66Yds4AlPguE9kvKiL7Bm78YJx0YQbj6I9t/g3WrxzDBgQ6uDt/AiNhoF3IOEA9Ht2l6yLrM4PnX4OHkvV8KZ6XnBCTubqspOaRUV5IT7xI+JWdNH5INYcVYphd0RgQT0T40K01vUVMPE/gZftLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nPRpfltP066J3isZV5OX4bHX/ZpjvXKITGjOzNWD6LM=; b=I8Ge6cLKD4QXyp4geGFRt7s6yhodIFkuWOzEGiGPWihBMJcD/FBZH6DRZopvo4zBOees4bG5TZLk+lw02Ehx6d24/Rwo1L1UwBD2dI4t6XM1MEENuRPZCkqlM7e2yp/O6AOqcH0Bxu1ZBY0mBC+vDr6BLbDgU4YdKDGLZv0szC7PQmrUDRx43OX9Bq5VjwwjjnACOC6BoEHePzmJpLPsGULcEzRwgalKj4dIJON5EjrS1a65etKd5+CjMeEtZp1ICYgFBqkLfaCUY+FwfMYyQp0d+pRwiQ0tlHlzGScmmlSwtvMB2xE/QYb5HH2qoU2C1gykkhfW8fI0V5jbTuo+rA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nPRpfltP066J3isZV5OX4bHX/ZpjvXKITGjOzNWD6LM=; b=Zzq2Com7wPZ2tRkCgEzrx3TUbH1GsboTTG48YrZsiIEXYe1JhlKjd+3fx1OUprlmcbJdBk5+IKxftXWnVCUr52tA0UWpCuUeGZ+z8TKlSMko7/gKdNiMTIjKL5w9k9byh19PiGNaOTyCdxHppLKoUWQh/nS54gVV03N5H1VI260= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by VI1PR0801MB1757.eurprd08.prod.outlook.com (2603:10a6:800:5a::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.17; Thu, 27 Jan 2022 04:06:49 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7c20:8c83:fc45:db99]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::7c20:8c83:fc45:db99%7]) with mapi id 15.20.4930.017; Thu, 27 Jan 2022 04:06:49 +0000 From: Honnappa Nagarahalli To: =?iso-8859-1?Q?Morten_Br=F8rup?= , Feifei Wang CC: "dev@dpdk.org" , nd , Honnappa Nagarahalli , nd Subject: RE: [RFC PATCH v1 0/4] Direct re-arming of buffers on receive side Thread-Topic: [RFC PATCH v1 0/4] Direct re-arming of buffers on receive side Thread-Index: Adf45cqRWsm3zOPbTBqjsnkimDKFQgBU+6HQBj0VyOA= Date: Thu, 27 Jan 2022 04:06:49 +0000 Message-ID: References: <20211224164613.32569-1-feifei.wang2@arm.com> <98CBD80474FA8B44BF855DF32C47DC35D86DAF@smartserver.smartshare.dk> In-Reply-To: <98CBD80474FA8B44BF855DF32C47DC35D86DAF@smartserver.smartshare.dk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 77FD39F0FC843449835C1DB2AA0E5490.0 x-checkrecipientchecked: true Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-MS-Office365-Filtering-Correlation-Id: b4bb95b1-cc17-407d-69fc-08d9e14a76bd x-ms-traffictypediagnostic: VI1PR0801MB1757:EE_|VE1EUR03FT064:EE_|HE1PR0802MB2588:EE_ X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:9508;OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 9KWeCH9iA67Vzzd/vOch/30RA6jMS04ckyMCNigmX63miuYBFU131UHzYeovKMGqozWbwC+25AWLeMxcuEbDRoYbZiUXGDiaOHGKdaluLBrEiSRBkgnVnlqlyWPEyGzHKm6vVDBfcSqyNBQF88rgXyfKNh9VXnwvCAV17pVhgYxjtxCkC3YQGDA6q6/VWf2UMQlKiNFtHS+NpkgjnOIYwXcHRsmp9x7AnG2SKs38/oImgb/463OIWH/zp4+U0i/yXU6WIc5IkCOYTeTg77edO2zS5GF5xTArHr5qZNWTScM5/SSgWStFN6/b5m8ZGG84N1VZjN21l/tf/ppJIA0eXRpt0gvygtT4Dph6ZNFr+rcQLt+V70b0DcHFnmUAVkYn825Ms2BY9CDMPY8PDYnkIQaLeLMAayVdcYzQSiSyLp2Od90Ixo+GlBruYIvssgLZtbiO8qCIgWWnaiivKhPk/o6TM4/EZhqLp/jTyck8BwUSfknUV/L0tHGrpgshQ8OFdCFlwpFyq7ERqFp68OL4juA42xvtT0QvjyKPqlVb082TdnJ/02zYPgKSzB0cJFFZEI9zbQJTrgLg29vNq9ddwltAIhK97QxAVJSLtzCxoDG6e0W3Jq18vqwT30ESMl9Y+pvoN6nc3S+vCW+uWLh73ml69hDhpof/AXaacGWiey08PG8lTLQuaVezP9d+Jwu4xrE5xdv4UpVU3UveOQdeUw== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(4636009)(366004)(71200400001)(7696005)(53546011)(6506007)(2906002)(66476007)(66574015)(26005)(186003)(66446008)(64756008)(66556008)(52536014)(508600001)(8676002)(4326008)(86362001)(83380400001)(55016003)(33656002)(8936002)(316002)(54906003)(38070700005)(110136005)(38100700002)(66946007)(5660300002)(76116006)(9686003)(6636002)(122000001)(20210929001); DIR:OUT; SFP:1101; Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0801MB1757 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT064.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: a1af202e-23f2-4ca8-e3b8-08d9e14a717a X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1u3AcT88wv/E4X4UfnF6wxJXFRRCs/zOcYuf/ZG2h++ajj/HzDU57i6sQQe8pUOJVEpvnavjtyh1X09HAjxDlQHjw+z6ZqJX2C32zScgivD9Umtfd1wWk55frR18TPh2l4KB3xp2M4agtGvb8VysYced8kTHgA6mAOauZVvlvOd/+lFeA2XcGqiHap8Fh6QSscr44eiWzBa+JYvRqCWbcCjsb1TjKlkL8x4AO85TSmLYp1l8L3byv7RgEqnQvbGFFYcBffj4DACllw8few2lQ0Z1YOmyx0yv7byPsARX6utQhs33RIiAmwgWDA9Plp/SvcjLM5PN8GYO37vl5HIxF/o7Rcb/Hnd9YVsCIbWXZSZ2/Y/qqt4OVosv8397PwIA4Olfxx/+CiG41eg20cesHm+m6qPVm3TLQN9az9/1yradAS47ItSweufi5FQ8eZvzHYcAAeBekuuhtPPjjB2JGZ97bp+IB5GobwOjSeoDsL4xyhcwPZa5UuITRUtLdWr7lXhCPcP80BpRhQE8rg2XTJxadACLDCL3W5XGv4nBo30DpGwxUmYqedclBNmXgjBbV8FvvaPfW2N1MbNiQqhuu12UcJVs1Y6GqdtTV+XOxUH6dSfaZUDs+xnc0UjRDUH2X84GudllrAw5VE397aQ4sH6T5+AVWZD+Wa+TYtjnHx1FADZtn3a3bGmSLpQRMdBBCTap3mRbw957yvigNHWcc7+hZUG7pjsW3+mvzTPAlPkNjcRWoVtlfoHu0FsWQ2t8392szk4bsUwYGk0nZAnrEg== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(46966006)(40470700004)(36840700001)(40460700003)(186003)(316002)(33656002)(70206006)(86362001)(2906002)(6636002)(356005)(66574015)(47076005)(110136005)(82310400004)(26005)(4326008)(8936002)(6506007)(7696005)(5660300002)(52536014)(8676002)(336012)(508600001)(81166007)(55016003)(9686003)(53546011)(36860700001)(83380400001)(54906003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jan 2022 04:06:58.4496 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b4bb95b1-cc17-407d-69fc-08d9e14a76bd X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT064.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0802MB2588 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Thanks Morten, appreciate your comments. Few responses inline. > -----Original Message----- > From: Morten Br=F8rup > Sent: Sunday, December 26, 2021 4:25 AM > To: Feifei Wang > Cc: dev@dpdk.org; nd > Subject: RE: [RFC PATCH v1 0/4] Direct re-arming of buffers on receive si= de >=20 > > From: Feifei Wang [mailto:feifei.wang2@arm.com] > > Sent: Friday, 24 December 2021 17.46 > > > > > > However, this solution poses several constraint: > > > > 1)The receive queue needs to know which transmit queue it should take > > the buffers from. The application logic decides which transmit port to > > use to send out the packets. In many use cases the NIC might have a > > single port ([1], [2], [3]), in which case a given transmit queue is > > always mapped to a single receive queue (1:1 Rx queue: Tx queue). This > > is easy to configure. > > > > If the NIC has 2 ports (there are several references), then we will > > have > > 1:2 (RX queue: TX queue) mapping which is still easy to configure. > > However, if this is generalized to 'N' ports, the configuration can be > > long. More over the PMD would have to scan a list of transmit queues > > to pull the buffers from. >=20 > I disagree with the description of this constraint. >=20 > As I understand it, it doesn't matter now many ports or queues are in a N= IC or > system. >=20 > The constraint is more narrow: >=20 > This patch requires that all packets ingressing on some port/queue must > egress on the specific port/queue that it has been configured to ream its > buffers from. I.e. an application cannot route packets between multiple p= orts > with this patch. Agree, this patch as is has this constraint. It is not a constraint that wo= uld apply for NICs with single port. The above text is describing some of t= he issues associated with generalizing the solution for N number of ports. = If N is small, the configuration is small and scanning should not be bad. >=20 > > > > >=20 > You are missing the fourth constraint: >=20 > 4) The application must transmit all received packets immediately, i.e. Q= oS > queueing and similar is prohibited. I do not understand this, can you please elaborate?. Even if there is QoS q= ueuing, there would be steady stream of packets being transmitted. These tr= ansmitted packets will fill the buffers on the RX side. >=20 > > >=20 > The patch provides a significant performance improvement, but I am > wondering if any real world applications exist that would use this. Only = a > "router on a stick" (i.e. a single-port router) comes to my mind, and tha= t is > probably sufficient to call it useful in the real world. Do you have any = other > examples to support the usefulness of this patch? SmartNIC is a clear and dominant use case, typically they have a single por= t for data plane traffic (dual ports are mostly for redundancy) This patch avoids good amount of store operations. The smaller CPUs found i= n SmartNICs have smaller store buffers which can become bottlenecks. Avoidi= ng the lcore cache saves valuable HW cache space. >=20 > Anyway, the patch doesn't do any harm if unused, and the only performance > cost is the "if (rxq->direct_rxrearm_enable)" branch in the Ethdev driver= . So I > don't oppose to it. >=20