From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2BE12A04DD; Tue, 10 Nov 2020 17:30:19 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5CE5A2B9D; Tue, 10 Nov 2020 17:30:17 +0100 (CET) Received: from hqnvemgate26.nvidia.com (hqnvemgate26.nvidia.com [216.228.121.65]) by dpdk.org (Postfix) with ESMTP id 39FC62B93 for ; Tue, 10 Nov 2020 17:30:14 +0100 (CET) Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Tue, 10 Nov 2020 08:30:16 -0800 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 10 Nov 2020 16:30:12 +0000 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.171) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 10 Nov 2020 16:30:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E5PLz7mrz6s+7Q0W7eu39nwp/APmKKlHPlxIN1+xbfipqp/nTz5qaIlkyqXwJDp69998FJJhf4OU18KwdcLuIPxekEnegKbyc6UqPG5+hcF/5Qxzan3LkC0i52xCrLhnJzRdm8R/mAvraufgainX59uasmRMKsTlZZ+UAqMWIt2H2VmeotBR6ArX4a2nw7fUYA/Zo1To+sdHw6ah/m89IqJ9hoNL60bmFNt5TzV38ONrIkzAtCWNjj0Y6EyoquBZNSI/SWLhMNXq/si/B9YaX5/s/Z4/yDmy9f9d9w1qcxsW46GK1ZY5T+pzVyAMBqDyok/7zBZcMhT8f7s5oju6cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5/K1TqWKrxV/qBQP+UxzXm3Kg8P5CPb67IDqlWFF1SU=; b=YuJO97Co7wSqEUC5rNn4eYm8iTUUeugS8wzxIiQ1ptMXwj7xhzUScHmZAV0zFA0vgvzkUcSlURkPer5/4HFCBvUHf/5mesgkA8aXLHdJ0z7cfkiypYULcXvuKaYTDYtRoHZe/TAsGUjdkyjD6odMweSHBSnLVcdFUwTkDXj1IC/qBzGiJpgr95OIQeRSEUjBgL6ZogWJ3lPLX8nvS0lJMXSCfmxNPgP5X1z6UZ8Y+JZeQI8Ro+56h6ROlQFrGdh3m9LYk4MF7eqn2Ars6UlCwTITM+oo3Ti64eTMhG/7wi6PKNgJRdLqaOo7C9+2csMkaGcjcfskgF1qJQdWabIKKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none Received: from MWHPR12MB1501.namprd12.prod.outlook.com (2603:10b6:301:f::18) by MWHPR12MB1805.namprd12.prod.outlook.com (2603:10b6:300:106::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.21; Tue, 10 Nov 2020 16:30:10 +0000 Received: from MWHPR12MB1501.namprd12.prod.outlook.com ([fe80::ec4d:f17d:402c:78b4]) by MWHPR12MB1501.namprd12.prod.outlook.com ([fe80::ec4d:f17d:402c:78b4%6]) with mapi id 15.20.3541.025; Tue, 10 Nov 2020 16:30:10 +0000 From: Slava Ovsiienko To: Alexander Kozyrev , "dev@dpdk.org" CC: Raslan Darawsheh , Matan Azrad Thread-Topic: [PATCH v2] net/mlx5: improve vMPRQ descriptors allocation locality Thread-Index: AQHWtYcNLQF1LonDEEu/t5jhJkS4iKnBkkgw Date: Tue, 10 Nov 2020 16:30:10 +0000 Message-ID: References: <20201106000442.26059-1-akozyrev@nvidia.com> <20201108042357.28542-1-akozyrev@nvidia.com> In-Reply-To: <20201108042357.28542-1-akozyrev@nvidia.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: nvidia.com; dkim=none (message not signed) header.d=none;nvidia.com; dmarc=none action=none header.from=nvidia.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 61c598f4-2a57-498f-b2d1-08d88595e4c6 x-ms-traffictypediagnostic: MWHPR12MB1805: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: nE6Mv0Oo7KUvLMBgTb0vTxoezojZs6hX9Vc4WElxFBwpQrLNJpJxsc9S11graf4zo7b1Eo4CK3AXuX1VUlkPksIBt89H0a7PQO9NEngdyqdT6Xjs56xv0eWkJ3V7XcRwpVE3NcjKWh8GKwXaR0Z5cXLS060qkFidpQ/OWZg9gqJ/KovfHLu4QKgiIzUdX8Bp98TEYofje7+/RuNrgt/131KghJXlehqaJjHGK+RIPn/vpg+qOXtdNXBHLrj8ZswYlgagIhy+NrsVpNgE+ileghQbBl9YnC/4FkgSmBa9QfT4kmLj3YtFffY4T0WnM/r+ilY8476GPA4UUIrmiSCvcg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR12MB1501.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(396003)(366004)(346002)(39860400002)(376002)(86362001)(66946007)(54906003)(52536014)(110136005)(5660300002)(26005)(76116006)(7696005)(478600001)(107886003)(66476007)(66556008)(66446008)(316002)(9686003)(6506007)(186003)(8936002)(53546011)(8676002)(33656002)(4326008)(83380400001)(71200400001)(2906002)(55016002)(64756008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: beuwJ6VpKwMzg40amXjg4JpN6mDoRjle+cHdaxdj12gvWsZgzpB6AcRu7aqijyOuA1SSjgLxnuUMYXYKMtC90swGoMHBXg70S2LRrGXZPY5QHtKJarz6Rp7VFox4QL4VtkPleLjtuMaFC0K/841KR5m+gDKpCI+mimkA5hgaAF1Uka2mIZ6OGvdvB6FVxGkp82WEvGdF34wwdpcmqxfPn35jb7JDfTfxdfvKwTHED52QxOcGFcNPefgRgnIK710WurR93hhAme4gB/ep3P1+N14i2FKbp5GFJ3ZRXobe9L8PmSf+e8uo4gTYb7FDqC4mn/OqID4nOjruf7+uHxnK1qoTzPEJxusDmaAkAQA58EhuY2UCFiVty7X6BcvwWFwB3OVpW6WN4F4MyOKJgFeluYfNIG8bLAKKTdtfI6YFkf1DMum8mgiBB03ysOdIwrqwdz64oZ+jCffeW2FDgt+9HdcfkEDQVHOf+wxesvr3MT3p/DEcRha5S/AxZq+Ia+lFsT7a2RR/Zv2rONfcHNFyNYFtR+3fRZO4s+DCmnMhIct230CfOMjpkbW2rMrKQXLbUoIYQu5ORoknhPyJAwY6c2zbo2Anlrdv59fz/R6m/b4Y8ibdEko5v0VIzIic+37K4yDaZWSByn4JF8g5e6n4wA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MWHPR12MB1501.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 61c598f4-2a57-498f-b2d1-08d88595e4c6 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2020 16:30:10.6144 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 28YE90gERj4qLQQCknsEO8TCdkTbj9MgHiZwukC2bv26DOklg3h8BaVDkPc+7dIX5T6pVpVO/qKJyru85313DA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1805 X-OriginatorOrg: Nvidia.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1605025816; bh=5/K1TqWKrxV/qBQP+UxzXm3Kg8P5CPb67IDqlWFF1SU=; h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:From:To: CC:Subject:Thread-Topic:Thread-Index:Date:Message-ID:References: In-Reply-To:Accept-Language:Content-Language:X-MS-Has-Attach: X-MS-TNEF-Correlator:authentication-results:x-originating-ip: x-ms-publictraffictype:x-ms-office365-filtering-correlation-id: x-ms-traffictypediagnostic:x-ms-exchange-transport-forked: x-microsoft-antispam-prvs:x-ms-oob-tlc-oobclassifiers: x-ms-exchange-senderadcheck:x-microsoft-antispam: x-microsoft-antispam-message-info:x-forefront-antispam-report: x-ms-exchange-antispam-messagedata:Content-Type: Content-Transfer-Encoding:MIME-Version: X-MS-Exchange-CrossTenant-AuthAs: X-MS-Exchange-CrossTenant-AuthSource: X-MS-Exchange-CrossTenant-Network-Message-Id: X-MS-Exchange-CrossTenant-originalarrivaltime: X-MS-Exchange-CrossTenant-fromentityheader: X-MS-Exchange-CrossTenant-id:X-MS-Exchange-CrossTenant-mailboxtype: X-MS-Exchange-CrossTenant-userprincipalname: X-MS-Exchange-Transport-CrossTenantHeadersStamped:X-OriginatorOrg; b=EMAxQHaOKzUonRCB3KgzJXdOkBFUXu+PENHZ1qw5pEI7luj4dQkIFwLNpShx/l6Le T/CocM1fONNtc5Sar0n8aWwxoGBzd6jW+PdH/jyBdi/Yx63E52K0Tz9XdFN4gZVdF6 NDZnRGJeLfVrFVn/lLw9aGqbEAy+fS+aiS18wIxs55Jbgb5pB+jCZTBekQ4tdsQPge StBuhQPlPz8diqh5SprJGodVgjOOL/hfqAyqY6led4fCPaq/67+JLQFb1GAoxlkq8b kkxWBq7U36IMxVxiozf7OauUXV4r+b78k4bDDMDaVmdj6wuQ4ou3Kp++3CNXHnEFpP pbOvVKhiLzJHg== Subject: Re: [dpdk-dev] [PATCH v2] net/mlx5: improve vMPRQ descriptors allocation locality X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Alexander Kozyrev > Sent: Sunday, November 8, 2020 6:24 > To: dev@dpdk.org > Cc: Raslan Darawsheh ; Matan Azrad > ; Slava Ovsiienko > Subject: [PATCH v2] net/mlx5: improve vMPRQ descriptors allocation locali= ty >=20 > There is a performance penalty for the replenish scheme used in vectorize= d Rx > burst for both MPRQ and SPRQ. > Mbuf elements are being filled at the end of the mbufs array and being > replenished at the beginning. That leads to an increase in cache misses a= nd the > performance drop. > The more Rx descriptors are used the worse the situation. >=20 > Change the allocation scheme for vectorized MPRQ Rx burst: > allocate new mbufs only when consumed mbufs are almost depleted (always > have one burst gap between allocated and consumed indices). Keeping a sma= ll > number of mbufs allocated improves cache locality and improves performanc= e > a lot. >=20 > Unfortunately, this approach cannot be applied to SPRQ Rx burst routine. = In > MPRQ Rx burst we simply copy packets from external MPRQ buffers or attach > these buffers to mbufs. > In SPRQ Rx burst we allow the NIC to fill mbufs for us. > Hence keeping a small number of allocated mbufs will limit NIC ability to= fill as > many buffers as possible. This fact offsets the advantage of better cache > locality. >=20 > Fixes: 0f20acbf5e ("net/mlx5: implement vectorized MPRQ burst") >=20 > Signed-off-by: Alexander Kozyrev Acked-by: Viacheslav Ovsiienko