From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D1052A0C49; Wed, 16 Jun 2021 22:26:20 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B511E4069C; Wed, 16 Jun 2021 22:26:20 +0200 (CEST) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2079.outbound.protection.outlook.com [40.107.21.79]) by mails.dpdk.org (Postfix) with ESMTP id B641B40683 for ; Wed, 16 Jun 2021 22:26:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Znxqhf8wOuUae8V89V3Gn2PoDU3icFi1l6MbyjoEwew=; b=wFeVKzOY/GWtimCKhhNP8Tt4fM2jVASy6Ie4TC3nj+9FTWIKiD8F2oAimKDSirVIrTAf9pGcM9NTqa5bQHIj+SnR+petjI63xhRCQezPj3om5NE1S/EJmk9ZZhGOZothvYLg30xB/8q9Ow/QjKworsyplZy8Fx0I9k4J5fCsUvU= Received: from DB6P195CA0011.EURP195.PROD.OUTLOOK.COM (2603:10a6:4:cb::21) by DBBPR08MB4363.eurprd08.prod.outlook.com (2603:10a6:10:ce::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.19; Wed, 16 Jun 2021 20:26:07 +0000 Received: from DB5EUR03FT003.eop-EUR03.prod.protection.outlook.com (2603:10a6:4:cb:cafe::7) by DB6P195CA0011.outlook.office365.com (2603:10a6:4:cb::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 20:26:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dpdk.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dpdk.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT003.mail.protection.outlook.com (10.152.20.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16 via Frontend Transport; Wed, 16 Jun 2021 20:26:07 +0000 Received: ("Tessian outbound d5fe3fdc5a40:v93"); Wed, 16 Jun 2021 20:26:07 +0000 X-CR-MTA-TID: 64aa7808 Received: from ac0c3f1afe75.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 71BF1057-E2D5-4E22-8452-620E8D47512B.1; Wed, 16 Jun 2021 20:26:01 +0000 Received: from EUR03-DB5-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id ac0c3f1afe75.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 16 Jun 2021 20:26:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j1UiY4W8lFBYJhrpoaaTrnReEcgbQYVFYfnz0+ruBR3thpb63HI3krXno8zGyQ0+FGncAR1OsBrL9sLnSoUMJ4NRtIwpoSXaj+E4jyHIPniL2ZCRBVTNhItgx+VMhxDdWlAX/WMmKx69r+EUqVpCXYzgR4xST1xVylBbpH6PmyQ5DhHgKSSSQnL7rOqDLmzxVFhsGi3By+ZS9WHKlNFMoswB2Cequed16lz76awuA66QjLouFu87uiTve0FDBeoAKMauuhT7ORUrYG/28mCofUlSx+i+oUwWVQs8Vs5jWdXb74iBNz707SwcQMJ/xMDijaxn8iOpN5+YKyRU8s/v3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Znxqhf8wOuUae8V89V3Gn2PoDU3icFi1l6MbyjoEwew=; b=bYjKJI0d5NWaB6CvaZxvSOLS6bpjvlvEkNgrRk6BOEqwu2rWCIB2EBIwdFxM2qbibZJQYUXwGkEDfHCzgE9y5xlvTE63ywXuPps87HcAQJOSZ12q/IqS8pJIEhkVHrlltuBtwAfv9ZRL6m0NYqo9TJMsWOaefgqCbJ4nU6AVIY/t1G0IruEdi+vhBQyGqDeu0OFZIyparpWwipGZTBwhdQnc8ODrxUY79Xccva2jRYEN6yak9QRofY6yBzAxpbKCQkQj0P4NxtV/x5BpwzbBvDf5+UZVPCIf6RpIo831NsVocSwvMpvA6eainR0oxW3GHIkgVzq5geZZbdkMouo7zQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Znxqhf8wOuUae8V89V3Gn2PoDU3icFi1l6MbyjoEwew=; b=wFeVKzOY/GWtimCKhhNP8Tt4fM2jVASy6Ie4TC3nj+9FTWIKiD8F2oAimKDSirVIrTAf9pGcM9NTqa5bQHIj+SnR+petjI63xhRCQezPj3om5NE1S/EJmk9ZZhGOZothvYLg30xB/8q9Ow/QjKworsyplZy8Fx0I9k4J5fCsUvU= Received: from DBAPR08MB5814.eurprd08.prod.outlook.com (2603:10a6:10:1b1::6) by DB6PR0802MB2150.eurprd08.prod.outlook.com (2603:10a6:4:85::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Wed, 16 Jun 2021 20:26:00 +0000 Received: from DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::f15f:821c:74c5:2482]) by DBAPR08MB5814.eurprd08.prod.outlook.com ([fe80::f15f:821c:74c5:2482%2]) with mapi id 15.20.4242.019; Wed, 16 Jun 2021 20:26:00 +0000 From: Honnappa Nagarahalli To: Bruce Richardson , "Zhang, Qi Z" CC: Joyce Kong , "Xing, Beilei" , Ruifeng Wang , "dev@dpdk.org" , nd , nd Thread-Topic: [dpdk-dev] [PATCH v1] net/i40e: remove the SMP barrier in HW scanning func Thread-Index: AQHXWt6+SyccfibsvU+3xAtLgQFgXKsHQPmggAFjrICAAEpKsIANwpEAgAACSwCAAG010A== Date: Wed, 16 Jun 2021 20:26:00 +0000 Message-ID: References: <20210604073405.14880-1-joyce.kong@arm.com> <561469a10f13450bae9e857f186b0123@intel.com> <2cb94e2e1bf74840acaadc389b4745f5@intel.com> <12226b6e56ad4c11845242031c9505d9@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ts-tracking-id: 64B6962DBC41DB4FB4C0D58DEE944F62.0 x-checkrecipientchecked: true Authentication-Results-Original: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=arm.com; x-originating-ip: [70.113.13.105] x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: c16df2fc-8934-4486-0da3-08d93104f902 x-ms-traffictypediagnostic: DB6PR0802MB2150:|DBBPR08MB4363: x-ms-exchange-transport-forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: B5rGt/WJU303Cn1am6VSZ+LAzbtrGjApx2mpNfK+O9lViAHktFOV17NkfSZcM09HUbHFolB+0q/LqxfMfucV+uECFJlblB3/817xViPboIkJmaMkNX7EZT1or7soUxqnfrSLsipmyVA8DXodHKbHrUSIB+kfCf+3zxuEi0cZ2XOhCS+JR0IphUwNWg8wEVxpSHaCWn+kT44ynqHzd1CRRB2sUCzP5UQ12i9l5bfDlMDXzdJ5sGFbLF1015iY/ZR2kS2SUddo6TfEpKOvsmt/zUQAQgv16NXyfjkEQMJMqOJEmQPk1jmi99INRKQWmNTtK7QMXeKq8Zp1M4Xmp6qdQL3XVGxNFPPJ16guhVWHUbhWtRLU/EfUEUojKLR/2mEwTzTrsaZlbh+KH3sk8c24aMjeOfKf3aTXEU8l78KDBZjPbGSPjnX+fx6C4sX1Lvyeun84vliohGCm1WTjFjMaeSgianAac0iT0KIJX3W1i0aIw4Vv0nQfgJb49vhDwx+Jgd/rLGVQfJahdXa3bkQVbRq7SjBrSkEk8N9/cclUC8AvatcJ06sM+W85PdjG68BeoJzneFIhB1NTwRIw6fNdRxPbmtA8s9VOLtY7H6M8Lus= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DBAPR08MB5814.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(346002)(376002)(366004)(39850400004)(66946007)(66476007)(64756008)(66556008)(66446008)(76116006)(4326008)(8936002)(478600001)(8676002)(2906002)(38100700002)(71200400001)(110136005)(54906003)(83380400001)(86362001)(316002)(33656002)(122000001)(186003)(5660300002)(26005)(55016002)(6506007)(9686003)(52536014)(7696005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Aw+eXDkvPLH+YtCshBWRBFt2Rpx+vF4O6/oESJ9j69D9O+YE+ZSiAtBmINOc?= =?us-ascii?Q?C6YCMxZ+xdh2qsVMCThf2QQuUJMWoHCyzJCqlLKjTWU+kL7i/N9nIoaTQ84w?= =?us-ascii?Q?Q2ybf6Xi4cvT0vMoyl/2uKuswO4vdMb8am2tYnPLWwprEQUCz5mWG1GxY804?= =?us-ascii?Q?crknpzR7TxwBXztk0DTvJOUEm63vBN9mhAEbLpVbmJLvBgkVf/RkyW3YrGIT?= =?us-ascii?Q?9uW7D66xE7KwvU7AbNgKDdT2dq0w3KsTtuhz/AUKndZTXJPaE14GHjLcj2Kh?= =?us-ascii?Q?LDTVLo5tQI+VdqDjWAEf05d1925kPhY5NFkIXgQueAJWYFXD04iFkGW9HRxK?= =?us-ascii?Q?ksjRmhUFNrXViSrif1P8xyZwLcFk+auNZg8rqHZA4/maDpA4Ju3qAv0/Z4z5?= =?us-ascii?Q?0iCP43TXb5xmpld/sJLQc04vvtJVlsaLWgNdBI1qXkUiMG0B/NOhYr4mERiR?= =?us-ascii?Q?R3oH9bbnPkeX15MOm7RAwMjKjni0KlIb/vo7DHCZNCD0P7bV99sEBDd86e7o?= =?us-ascii?Q?r51BXXk7c5hsCJHRj5OFbLhG40BTpeDZLyNu2sCcF+og2C/4Lpa62RQDr8yb?= =?us-ascii?Q?KFEzCIRAC1w9jLxZnHFibPi7Q43+I8d+oMLiPXm1bCuZdrMQgXpfQ/NXWGV7?= =?us-ascii?Q?n9WQj2GjKe9e4NV/n+h/8xHQsjAj27qjdzYCS/DKJL9iagq0aTdxbVhmd4A2?= =?us-ascii?Q?SqXwSrGsHqMLTmVcDsa1yityQnMNbCAH9EXv8Ht+VRlvlYMsBKh8gN5fXPoW?= =?us-ascii?Q?HIFtBJuouj04bVvG4C8MZpKS9MN//NvJTsgzUJvKlhl1K1WAsuB+3hkx1Cmp?= =?us-ascii?Q?yg0H73bTuhwHfRM1VpkEEbxMfJBbwvrY6ED0lOlhI2cMlbfERYWFtAUk+ImG?= =?us-ascii?Q?lKP6ZbfeDpHTFZbhzWo8or99qV0BvApowFU0k+LqTWA78jwwOJXCLjsEhKYF?= =?us-ascii?Q?uZ01PdXgSlj6fOSXmLawXz/6BXbTZOvhVxArfJcCNWupUoh2F87VgdGMQiCp?= =?us-ascii?Q?eHv9I1wCIV5sVTvs0b3JrVBdltxSuCMjEAeqiE2Ydz0XTqbWSguweMcwMYAY?= =?us-ascii?Q?j7HwQp1pm+N33ElXDyiGFlu14UGIMwR4ua0d7s7JDnghEzDXuYJ8jziWCtWd?= =?us-ascii?Q?jD4MvEBLkRMSywUx/nVUtro8MKLlXUuIns3ibJDVnFep+MO392QyX5M4z29k?= =?us-ascii?Q?vnSqsZ4Z/plfxIhdEMFO8xIZZebiJBhFQrzEXh03MJU2jkP2VdglWSK7qOKK?= =?us-ascii?Q?0rW6sP+9c3hRxlIdWt0UhRfowTDvb06l9r2YOjx+i/CDBNOAEUc2nFqLBpGZ?= =?us-ascii?Q?kTmx295r8EO2Mvu1iud978xi?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB6PR0802MB2150 Original-Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 763b80b0-b200-4dbc-4a9a-08d93104f4f5 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: monzWwwah1TmMFv9/hFX8GYtm8ZqHt4J6lGfmtptxJ/pbjpVu8i20M9SZXS1VNUc6yxme4S+DB2l2pzd94cReVWiAnNq9mnYKtd5dX7KvH3SgWPl+MmyrSX4Gpg9+UAbaRmY5/uubvZdp4MkP9FUOIISJ6p4V8HOH0/rJKP+x1cZXBeUUUZQdQFJwniNEt87ZBp6QdxkqYwNU5r2XH3VipLIZXRS+7iNv1hbLpcHdKSECwwRfIO86ri+sGd9rTWenTVSKhXhu+m4Z0hO+DyzbuRfn/T/8HEFnhD6U7lmp1mtPhkj/3ymfqxuGGto20qvBl9+CVqI+CQHGrdafwH5JdXEd/bMGHnugMKWR3ofhX2KLeF+9iGUS5fmhjIt2/FOmZk3Y+3Ld4pB68x+LvOoeBU73oJhlIxpkMNIJePq2qtajKNW6uFUKQ7ZXoe1pkhslbk8k5wxFU0Zvz77oO3h5+AIELeqvlgPazAMnZD+N0P/b2U+Av1Ae1/HbL0PlcNiULWv3Wnm4SmMcMCt1JsC+dV+YhnhWrBys/Lu+uv+uHl9g0gPN0s6RmbR41yuc/V07w88+WM8DvYIr2A3T8O1XqU34UbxdNbRTYZTmudExPf42fr1ILmfAoVvdX8EQIPSz8Y7jXcY8AqZ2rSbNCt8MB7u+t0aly+ZdJe7IUDGnMc= X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(396003)(39850400004)(376002)(346002)(136003)(46966006)(36840700001)(55016002)(82740400003)(8676002)(478600001)(33656002)(8936002)(52536014)(356005)(4326008)(5660300002)(81166007)(70206006)(54906003)(82310400003)(86362001)(6506007)(186003)(9686003)(2906002)(70586007)(7696005)(83380400001)(316002)(47076005)(36860700001)(26005)(336012)(110136005); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2021 20:26:07.6837 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c16df2fc-8934-4486-0da3-08d93104f902 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR08MB4363 Subject: Re: [dpdk-dev] [PATCH v1] net/i40e: remove the SMP barrier in HW scanning func X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > > > > > > > > > > > > > > Add the logic to determine how many DD bits have been set > > > > > > > for contiguous packets, for removing the SMP barrier while re= ading > descs. > > > > > > > > > > > > I didn't understand this. > > > > > > The current logic already guarantee the read out DD bits are > > > > > > from continue packets, as it read Rx descriptor in a reversed > > > > > > order from the > > > > ring. > > > > > Qi, the comments in the code mention that there is a race > > > > > condition if the descriptors are not read in the reverse order. > > > > > But, they do not mention what the race condition is and how it ca= n > occur. > > > > > Appreciate if you could explain that. > > > > > > > > The Race condition happens between the NIC and CPU, if write and > > > > read DD bit in the same order, there might be a hole (e.g. 1011) > > > > with the reverse read order, we make sure no more "1" after the fir= st "0" > > > > as the read address are declared as volatile, compiler will not > > > > re-ordered them. > > > My understanding is that > > > > > > 1) the NIC will write an entire cache line of descriptors to memory > "atomically" > > > (i.e. the entire cache line is visible to the CPU at once) if there > > > are enough descriptors ready to fill one cache line. > > > 2) But, if there are not enough descriptors ready (because for ex: > > > there is not enough traffic), then it might write partial cache lines= . > > > > Yes, for example a cache line contains 4 x16 bytes descriptors and it i= s > possible we get 1 1 1 0 for DD bit at some moment. > > > > > > > > Please correct me if I am wrong. > > > > > > For #1, I do not think it matters if we read the descriptors in > > > reverse order or not as the cache line is written atomically. > > > > I think below cases may happens if we don't read in reserve order. > > > > 1. CPU get first cache line as 1 1 1 0 in a loop 2. new packets coming > > and NIC append last 1 to the first cache and a new cache line with 1 1 = 1 1. > > 3. CPU continue new cache line with 1 1 1 1 in the same loop, but the l= ast 1 > of first cache line is missed, so finally it get 1 1 1 0 1 1 1 1. > > >=20 > The one-sentence answer here is: when two entities are moving along a lin= e in > the same direction - like two runners in a race - then they can pass each= other > multiple times as each goes slower or faster at any point in time, wherea= s if > they are moving in opposite directions there will only ever be one cross-= over > point no matter how the speed of each changes. >=20 > In the case of NIC and software this fact means that there will always be= a > clear cross-over point from DD set to not-set. Thanks Bruce, that is a great analogy to describe the problem assuming that= the reads are actually happening in the program order. On Arm platform, even though the program is reading in reverse order, the r= eads might get executed in any random order. We have 2 solutions here: 1) Enforced the order with barriers or 2) Only process descriptors with contiguous DD bits set >=20 > > > > > For #1, if we read in reverse order, does it make sense to not check > > > the DD bits of descriptors that are earlier in the order once we > > > encounter a descriptor that has its DD bit set? This is because NIC u= pdates > the descriptors in order. > > > > I think the answer is yes, when we met the first DD bit, we should able= to > calculated the exact number base on the index, but not sure how much > performance gain. > > > The other factors here are: > 1. The driver does not do a straight read of all 32 DD bits in one go, ra= ther it > does 8 at a time and aborts at the end of a set of 8 if not all are valid= . > 2. For any that are set, we have to read the descriptor anyway to get the > packet data out of it, so in the shortcut case of the last descriptor bei= ng set, > we still have to read the other 7 anyway, and DD comes for free as part o= f it. > 3. Blindly reading 8 at a time reduces the branching to just a single dec= ision > point at the end of each set of 8, reducing possible branch mispredicts. Agree. I think there is another requirement. The other words in the descriptor sho= uld be read only after reading the word containing the DD bit. On x86, the program order takes care of this (although compiler barrier is = required). On Arm, this needs to be taken care explicitly using barriers.