From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10051.outbound.protection.outlook.com [40.107.1.51]) by dpdk.org (Postfix) with ESMTP id 0FF5C49E0 for ; Fri, 9 Nov 2018 01:47:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector1-arm-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0F7Ny+BbXI9lerZ8DYhZRRCH9WxrDg7jQmvlDtoFnuc=; b=ipneShfHq9VrjgZ28AbsTjbaC5SLiaWF+SQ2C0wbpa/KNjX0nHHOxrIHyrRcppCnYmtMFWulJc3LFM1Jm5cVyZ7KNgsbbxb12kFOLgKT8KEJA4GMDBQ441j41lgmWvTdMYi7bojXd0NCaa3JfgffgfRjwkJoDyfcFCLVWal8YGk= Received: from AM6PR08MB3672.eurprd08.prod.outlook.com (20.177.115.29) by AM6PR08MB3351.eurprd08.prod.outlook.com (52.135.165.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.27; Fri, 9 Nov 2018 00:47:56 +0000 Received: from AM6PR08MB3672.eurprd08.prod.outlook.com ([fe80::c1a0:51bf:cd33:2b27]) by AM6PR08MB3672.eurprd08.prod.outlook.com ([fe80::c1a0:51bf:cd33:2b27%6]) with mapi id 15.20.1294.034; Fri, 9 Nov 2018 00:47:56 +0000 From: Honnappa Nagarahalli To: "Wang, Yipeng1" , Jerin Jacob CC: "Richardson, Bruce" , "De Lara Guarch, Pablo" , "dev@dpdk.org" , Dharmik Thakkar , "Gavin Hu (Arm Technology China)" , nd , "thomas@monjalon.net" , "Yigit, Ferruh" , "hemant.agrawal@nxp.com" , "chaozhu@linux.vnet.ibm.com" , nd , "Gobriel, Sameh" , nd Thread-Topic: [dpdk-dev] [PATCH v7 4/5] hash: add lock-free read-write concurrency Thread-Index: AQHUbO4hC1hG0V5Pe0K6GLHhoSzYDKU9/ZaAgAA/xYCAA9sPQIABjUCAgALgE9A= Date: Fri, 9 Nov 2018 00:47:55 +0000 Message-ID: References: <1540532253-112591-1-git-send-email-honnappa.nagarahalli@arm.com> <1540532253-112591-5-git-send-email-honnappa.nagarahalli@arm.com> <20181103115240.GA3608@jerin> <20181103154039.GA25488@jerin> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [217.140.111.135] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM6PR08MB3351; 6:NI0haIU7FUBcPO3DaYfyYxz+L6dcFzbGMQAjf1iU/qUeOZlzJUFnwbX8+nGgbZXOyEvsQ3Y0tOLVVTIaMCdQCZxW/kqtnJJHX8typEVfH51GQ565AbRqlmUrwMdqQr5nBsxhoRsDiHTzwQePUtWQENPA/b59x05uGJraTYr2XivjGmvtyYt1pZbN951FnhOV9sMYps8LEJXl7od73RM+IbNnh2ouEhSMwAWIZDafzDgFQ0o7UdzQAiEHbXdjje1snda9MIPWTiK8mlvO1IZ1pjwYScTw3EF0TW6gA6g5oqD0/PTx9zURylo0xG11B3cICp0yllxUVJcR1spDkKrPiiBF/MI8HVeuzpAHFxV56hEuz7lje0OwVLS876InGuzdxbQDTotrb5OhgHe6aRxdCred5fTyiuQqVSaJHXERsOqllWEnNGJDNeg9jaOfiHjlsqvYfSlvA2/pM/G7O6X4NA==; 5:HOcStI+ZtnmicT1ye9gbDrY7MfEN7o1JZlEhcrLRyxO7HBHvK6mjalaxfkhh+vrlVAbP/Wqvpcvc7QkcgTMe+4UpXnIrwGMZH5R+V6OsXb4l+KrmocwYf4QNloHyxLASj52Ju1yRZdKZFoP/siiWlm8Ys9vDlxUH9aFIKXq59xM=; 7:J3FC50rycI6pR60p/yYhD5j6Yu52MfeGg5frtcsnZRdFJLdmKt7xeXW6FVVRcN+pRBXp844+UuEef3f5xnUDrylOG5JBMDOWEpphV+lSM4EYBW7mxNUNwL/GJYRA3znJBImGC0HYLxQicKiHpwlv9Q== x-ms-exchange-antispam-srfa-diagnostics: SOS;SOR; x-ms-office365-filtering-correlation-id: e562a75e-6d9a-41fa-28bb-08d645dcfd15 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM6PR08MB3351; x-ms-traffictypediagnostic: AM6PR08MB3351: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Honnappa.Nagarahalli@arm.com; nodisclaimer: True x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(3231382)(944501410)(52105095)(10201501046)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123564045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(201708071742011)(7699051)(76991095); SRVR:AM6PR08MB3351; BCL:0; PCL:0; RULEID:; SRVR:AM6PR08MB3351; x-forefront-prvs: 08512C5403 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(346002)(396003)(136003)(366004)(39860400002)(189003)(199004)(25786009)(81156014)(81166006)(8676002)(4326008)(2906002)(97736004)(6436002)(345774005)(93886005)(53936002)(14454004)(9686003)(33656002)(6246003)(55016002)(7736002)(66066001)(6506007)(102836004)(478600001)(7696005)(76176011)(72206003)(305945005)(186003)(256004)(99286004)(486006)(3846002)(6116002)(476003)(446003)(2900100001)(86362001)(71190400001)(26005)(71200400001)(105586002)(8936002)(68736007)(110136005)(316002)(5660300001)(54906003)(74316002)(106356001)(7416002)(11346002)(229853002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR08MB3351; H:AM6PR08MB3672.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: arm.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: hV65B6LxCvPXbY1LEOvBe9Sucq/T/60MT/QcOqgSBzPioozxSQGng4wvFiYoaW/cKNME4TY1cnt87ENqHb0gzcKxIS4TTnam95M2HLvlR2y7mALsz9N9yapLuIO+oRLVl9vRx5f2PQpKjhH8lkEXJz7x/iLEL7dWl8ldOkS1ReY6PBJiOltaPdEdDufXOSu0+yduXF2ZpQsopPIvrBm4lJh59XrSVhKcXWIGR0J5Ssgm9smAdNOSDizmPUw+54ZHB31FaCH6cX8MPeyNMyaQMeUMzTV+IrBeJIEKNLmhVz89khdft1cnXCLqv2lnKNyP/4i1nBRLvpOl0vbI6U1vI2rMk4FyE/nYxFIHY6UA8qo= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-Network-Message-Id: e562a75e-6d9a-41fa-28bb-08d645dcfd15 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Nov 2018 00:47:55.9817 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3351 Subject: Re: [dpdk-dev] [PATCH v7 4/5] hash: add lock-free read-write concurrency X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Nov 2018 00:47:58 -0000 > >> > > >> > 9) Does anyone else facing this problem? > >Any data on x86? > > > [Wang, Yipeng] > I tried Jerin's tests on x86. So by default l3fwd on x86 will use lookup_= bulk > and SIMD instruction so there is no obvious throughput drop on both hit > and miss cases (for hit case, there is about 2.5% drop though). Do you mean, if the test case has 'hit only' lookups, there is 2.5% drop? >=20 > I manually changed l3fwd to do single packet lookup instead of bulk. For= hit > case there is no throughput drop. > For miss case, there is 10% throughput drop. >=20 > I dig into it, as expected, atomic load indeed translates to regular mov = on > x86. > But since the reordering of the instruction, the compiler(gcc 5.4) cannot > unroll the for loop to a switch-case like assembly as before. > So I believe the reason of performance drops on x86 is because compiler > cannot optimize the code as well as previously. Thank you. This makes sense. > I guess this is totally different reason from why your performance drop o= n > non-TSO machine. On non-TSO machine, probably the excessive number of > atomic load causes a lot of overhead. >=20 > A quick fix I found useful on x86 is to read all index together. I am no = expert > on the use of atomic intinsics, but I assume By adding a fence should sti= ll > maintain the correct ordering? > - uint32_t key_idx; > + uint32_t key_idx[RTE_HASH_BUCKET_ENTRIES]; > void *pdata; > struct rte_hash_key *k, *keys =3D h->key_store; >=20 > + memcpy(key_idx, bkt->key_idx, 4 * RTE_HASH_BUCKET_ENTRIES); > + __atomic_thread_fence(__ATOMIC_ACQUIRE); > + > for (i =3D 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { > - key_idx =3D __atomic_load_n(&bkt->key_idx[i], > - __ATOMIC_ACQUIRE); > - if (bkt->sig_current[i] =3D=3D sig && key_idx !=3D EMPTY_= SLOT) { > + if (bkt->sig_current[i] =3D=3D sig && key_idx[i] !=3D > + EMPTY_SLOT){ Thank you for your suggestion. I tried it on Arm platforms, unfortunately i= t did not help. However, the idea of reducing the number of memory ordering= s addresses the problem. I worked on a hacked patch for the last couple of = days. I have tested it with L3FWD data set, it provides good benefits. I ha= ve sent it to you and Jerin. Any feedback will be helpful. >=20 > Yipeng