From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1E417A0C4B; Fri, 15 Oct 2021 11:12:22 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9ECCB40692; Fri, 15 Oct 2021 11:12:21 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 756AB40041 for ; Fri, 15 Oct 2021 11:12:19 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10137"; a="215043849" X-IronPort-AV: E=Sophos;i="5.85,375,1624345200"; d="scan'208";a="215043849" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2021 02:12:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,375,1624345200"; d="scan'208";a="571705924" Received: from orsmsx604.amr.corp.intel.com ([10.22.229.17]) by fmsmga002.fm.intel.com with ESMTP; 15 Oct 2021 02:12:08 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Fri, 15 Oct 2021 02:12:08 -0700 Received: from orsmsx606.amr.corp.intel.com (10.22.229.19) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Fri, 15 Oct 2021 02:12:08 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12 via Frontend Transport; Fri, 15 Oct 2021 02:12:08 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.176) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.12; Fri, 15 Oct 2021 02:12:07 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eKa8ia9a+lZ1AlL+7NXjRgnqZ0KXOGT970fwLAv8OF3xyoAJ8XfhvOwc7OaZG3Ot2krd7WVmjFFGTv1NxzNN+WphquxmfLzK7UwBr6kUbJJ98y4BhUJS1Llbe26hpC9Z3zXAMPNErLN6Ae5dWE5EzLnIsOj2yeae4pEEygpmSa6jB/qjv3jrspU31SsehZcswlnlKXPbMo/BJJ0Y73B8v1pjyEDO6Z0yn/jYs/d6M9eCfNiqefpJPlJeaeUmaT0ZOlSq4xgaWGuO0AVVmOj2f4fDUzkR8wsx+LtS54UJEWL2W0D2hxb3ck6dYzwRBgFpRyi7bDwJxNfaQ+vbNVXNAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J6QGF/8F25uyy/OpbD45SG/HJW4j6DbBex3v2LksLWI=; b=OyFMqb/RWagNaK59Uy8E08ArVFA+7+PduwvzmsBPI4jvV6DI/HXH2+6Vdmx94pflDJAbnsmQvF6Jmg/jw2xURLZAv2NIy08C/oQ0eNE7Ksr6TM4Q/IAlV5ORQMGBUd17FBAE88OhHJtXGV66veU/CQjlevdRKgNSdCPbm2Vo5N+VYbpi6107fppnU4UvID8U90Pa6L5o4kfEWlJhSTPKsgBroNqDwRt/a1vzr2uX7tvdN9TV+M5dM1e1WpzZVh9gDnB0slm6vsGIDT7oV7RDWJe4jz7cjSwUXEvy6ZkXGa5VwoirVYixDnRD6E4PkjknJdW8jD7iK/iYwxih2GnO0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J6QGF/8F25uyy/OpbD45SG/HJW4j6DbBex3v2LksLWI=; b=Z8pDls2d8TGnDWkCg+OYegPf95MncMxSu6UJkX0w/1RcPgSi1B2VL/cHqSZRfkCS3Yw4v4Mny/pRBO7hnyFYkAoW9ih2EOa0fGvjHy/om6U5u57iFYdZYomU7cUIkc4vTIC8in1UYRH4FJLxgyngnhlvwZk0cN0E1QzoprCh/5c= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5013.namprd11.prod.outlook.com (2603:10b6:510:30::21) by PH0PR11MB5190.namprd11.prod.outlook.com (2603:10b6:510:3c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.15; Fri, 15 Oct 2021 09:12:04 +0000 Received: from PH0PR11MB5013.namprd11.prod.outlook.com ([fe80::4c6a:b2b:c561:4c85]) by PH0PR11MB5013.namprd11.prod.outlook.com ([fe80::4c6a:b2b:c561:4c85%8]) with mapi id 15.20.4587.026; Fri, 15 Oct 2021 09:12:04 +0000 To: "Ananyev, Konstantin" , "dev@dpdk.org" CC: "Chilikin, Andrey" , "Wang, Yipeng1" , "Gobriel, Sameh" , "Richardson, Bruce" , "Mcnamara, John" References: <1630944239-363648-1-git-send-email-vladimir.medvedkin@intel.com> <1630944239-363648-2-git-send-email-vladimir.medvedkin@intel.com> From: "Medvedkin, Vladimir" Message-ID: Date: Fri, 15 Oct 2021 11:11:42 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.14.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: DB9PR06CA0003.eurprd06.prod.outlook.com (2603:10a6:10:1db::8) To PH0PR11MB5013.namprd11.prod.outlook.com (2603:10b6:510:30::21) MIME-Version: 1.0 Received: from [192.55.55.59] (192.55.55.59) by DB9PR06CA0003.eurprd06.prod.outlook.com (2603:10a6:10:1db::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4608.17 via Frontend Transport; Fri, 15 Oct 2021 09:11:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6936acf7-7b15-4e12-8a8e-08d98fbbdb02 X-MS-TrafficTypeDiagnostic: PH0PR11MB5190: X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:418; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9EkI8X4mIhT5gZ45FTFtiIQjFUi046Ti0lX0cbcQQCOuKWkubcZub2SMRclH+6GIbXF382pEOy0ElvJhvG4ysX4HCre0GCU8mP2shPGNkPeXRYpvekFYBgJMLI+qMxxiWSkbqSGOULHxIDpLvaI2ky7o0sDX5WncZ2LP+gp0fG0mFxdEseS4nP9p5HQiVbDTUz8oCNwUcZwHAuw0Tjv3Fz002o6Fjx/TdH2rHTKuuUHMCv4Uta/voO6abp5zVYgLBtpEPbCb9dIEp4BV9OZ0kjNz25OtRyWb/QSfugyeWbfkPLLuJKpL0Q1RjedKUltJ1+mtkHdsvKG0e4kNvsP2REuXhgTaIbFjMNJCkW0hezbtyd0n5TZFyrRkfVrAwywVPgEtmywRLJEafH+BpfpW0r1UyPLjAxENrfT7lECnCFsSFpLOY1LSCUo0S1KUIi1kmd4VERLPF5XD4zkvLp4yoqp6OmB4Jg05Yh4Kk6v0ullFIN9juaavwKR3dLeC7CScp3zi9XYdo2wgeWyjomjZixmViAxQGx7fAe1ebYLUd6vZFBYdh15c8B1WMsScolXOSLXbLq05PCZ3cvrfrYq8BBPtPA+XL85bA223K2xyd0chAGt9nILU5TyyrT/up5CK9gHUPX0WehMhhEwAhYtpgMfb5EUj9xd48+ge1xi8PnSSnFs111bhf/wOvNfEVDGG0mmLXZe/798ffvl7eL3OJO06dnRST6r14HB/uNWcIu7oX8UcUGUuGojWkX0GtCyJFzutANOVLmSzPCVYSRFz3A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5013.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(6706004)(83380400001)(508600001)(956004)(186003)(8676002)(53546011)(4326008)(5660300002)(36756003)(2616005)(6486002)(26005)(8936002)(16576012)(6666004)(110136005)(31686004)(38100700002)(82960400001)(316002)(30864003)(54906003)(107886003)(86362001)(66556008)(31696002)(2906002)(66476007)(66946007)(3940600001)(45980500001)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Q3p5RUprZGl3TTBSc2M3ZjNCT1dWbG9rYjNlYi9ySXBlL09ITzEzSGtocEFW?= =?utf-8?B?eGtlSFZJY21NNFZBbEhGQVAyejJJTXRjNEtOYTlKdS91Yk8vOHErRVZaaFl3?= =?utf-8?B?OGJ1eGh2RFQ0Qi9IV0grV1dyY0dEdmhSVDBZbTB2UGV5ZExjN3Y4c3R6bWJO?= =?utf-8?B?ZGMwYkhmWm93N3YzZ1lhV0I3cUlYYVFNSkZLTnNyTjY2YWcvakFTYklEakdU?= =?utf-8?B?WnpFZEx0RGE2WTBxTzNBajluSU9tamJodXNHb0dvNlF2dFl6VGJtT2hEWU45?= =?utf-8?B?Y2Z6bEFhYkdvaDRrTHVrNFZuVDFDR1duVnN1OW8xejhyVDh3bmlCZ2llcFM0?= =?utf-8?B?VytSMkcwSkdMQllkSnlOVmVKZk55ZzFDc1FYa25aSWo5M1ovc1VnZDAvTHBj?= =?utf-8?B?NTdGcUM0bnFodEdIVVArU1NwUWN3ZFlCVlk0L3IyQnhXK0NJd0gyTFJOTVN3?= =?utf-8?B?aUdUc3JERk83b2VFSCtiSUtGU0xDMkdHVTdsczVwNXB6M0tPSm1EbHRsVjlE?= =?utf-8?B?L1JoN3EyM2FVZFZBam9EVFFQM3kvdnlzU1l5YVY3aGxhT2trZzA5ajA1WmpT?= =?utf-8?B?bk9vZlpsZzBPSHZCOXY0VHYwVUI3cW5BRC9TekEwOTFRV3EzajJpTk5VY01r?= =?utf-8?B?UmxVV2FWRzZmOWpvbFZnTC81a3pjdVhVSDlPbUFDZHVLaWZHWEVGTkh5Q29p?= =?utf-8?B?TXRkdGdvZG9NR0phRmtvM3FkakNNY2JZSkxtVGxESk5yY3dDa3kxbFJXWmZ4?= =?utf-8?B?MmdMWWRSMDJuV0lFaEt2Mkw0SUx6Ujd0WHFmNHZpSnVXUk5BU3JCb0llYW1l?= =?utf-8?B?VHhmN3VaRFJiRUFBK2pGUHFKNGovajNmK2ExOG5JTXBiMEQwb2V5dzZOUUxH?= =?utf-8?B?alcrYzAzS2lKMkZBdFQ2NXZPUUhxbVFvZW15MzE4Z1pLSjV3QXRnYnlvVXdI?= =?utf-8?B?cE9BaWU5UFA0bnVtLzcxRHUxYmV2OFVxcDl3L0FhdEtoMEU4OGhDbFBKc1Zz?= =?utf-8?B?SWIydW8zb201MVpLNEpPVXRXRGJOdEk3Yi9KVmlNdkplNURXQ0dmQU9IdFRM?= =?utf-8?B?ZXV1NlpPQjNoQ3Q5NUdkTjBoSGRoY0FTaEhDSUlvcll6eEM3Q2NPZDZlNWgw?= =?utf-8?B?aEc3UXNTQXVoaWxJWWRVQ0FDbzFHZTk0M0o1ZlNzaEtCSDlrblljZG8reGVF?= =?utf-8?B?MWFQdnN4WVlwNUV5WFR6WnlFaXZlUUg4c2dJeXBDMzNXbml5VTM2OHd3d3Qw?= =?utf-8?B?c1VxM04zSG1aTnRtRDR5MVFWOVI2K3RxSVB0MFh3dFFsRFBTcW9Kd1pFU3NY?= =?utf-8?B?WTVqZGVjSnJLOUtHNXlDOU1PRk00UWx6ejZUMmlTSFlEdWJ6ZmRKUWVmTkUz?= =?utf-8?B?OHV3OHRLWXNmY1hDb1hNbFkxenhQYktCUlNWeStVdWxFZStMRFdDM2hDL2Ry?= =?utf-8?B?WFVEaWlqZndTNnNsNitteHY1NGRoRGJnb1FPT0RGUjdYbVlqdTExWldoU2l5?= =?utf-8?B?ZHBnWkJKQUFRcklmSGM4V1VkRlVKQ1VURUxuNkRrUnBBU1Q2NUtESVlqMHd2?= =?utf-8?B?L3dIa2xhVTdvSTRkaGNzNWFabXVEYzVBNTdkU2k5V0ErY0lzazFobDhjWExu?= =?utf-8?B?MkFkTE94NFhzTHlHTlIyU3Vicm56dWl3ekh2VTd6MVFZb3A2UndHd2ZuV3lj?= =?utf-8?B?cTQrUGlhd1k0NTZ5aEl0TWlKR0FLV0d2R1oveGdhVzF4cWZRdUxUUk5ST0V1?= =?utf-8?Q?N2yMH7tYZrYGeFs7LMEgj6JA5yHfcgwhTaGsbb0?= X-MS-Exchange-CrossTenant-Network-Message-Id: 6936acf7-7b15-4e12-8a8e-08d98fbbdb02 X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5013.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2021 09:12:04.7411 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SSmiGHO3VzDNOoNZzU5R3NworfrLIADtRXslseCReTWsuEhoUSST4Nl1pAMskGSkxl1EAkDrxHEkciLL50FPjbQ3BBIC7xz3LEa+WVs80io= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5190 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH 1/5] hash: add new toeplitz hash implementation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Konstantin, Thanks for the review, On 07/10/2021 20:23, Ananyev, Konstantin wrote: > >> This patch add a new Toeplitz hash implementation using >> Galios Fields New Instructions (GFNI). >> >> Signed-off-by: Vladimir Medvedkin >> --- >> doc/api/doxy-api-index.md | 1 + >> lib/hash/meson.build | 1 + >> lib/hash/rte_thash.c | 26 ++++++ >> lib/hash/rte_thash.h | 22 +++++ >> lib/hash/rte_thash_gfni.h | 229 ++++++++++++++++++++++++++++++++++++++++++++++ >> lib/hash/version.map | 2 + >> 6 files changed, 281 insertions(+) >> create mode 100644 lib/hash/rte_thash_gfni.h >> >> diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md >> index 1992107..7549477 100644 >> --- a/doc/api/doxy-api-index.md >> +++ b/doc/api/doxy-api-index.md >> @@ -139,6 +139,7 @@ The public API headers are grouped by topics: >> [hash] (@ref rte_hash.h), >> [jhash] (@ref rte_jhash.h), >> [thash] (@ref rte_thash.h), >> + [thash_gfni] (@ref rte_thash_gfni.h), >> [FBK hash] (@ref rte_fbk_hash.h), >> [CRC hash] (@ref rte_hash_crc.h) >> >> diff --git a/lib/hash/meson.build b/lib/hash/meson.build >> index 9bc5ef9..40444ac 100644 >> --- a/lib/hash/meson.build >> +++ b/lib/hash/meson.build >> @@ -7,6 +7,7 @@ headers = files( >> 'rte_hash.h', >> 'rte_jhash.h', >> 'rte_thash.h', >> + 'rte_thash_gfni.h', >> ) >> indirect_headers += files('rte_crc_arm64.h') >> >> diff --git a/lib/hash/rte_thash.c b/lib/hash/rte_thash.c >> index d5a95a6..07447f7 100644 >> --- a/lib/hash/rte_thash.c >> +++ b/lib/hash/rte_thash.c >> @@ -11,6 +11,7 @@ >> #include >> #include >> #include >> +#include >> >> #define THASH_NAME_LEN 64 >> #define TOEPLITZ_HASH_LEN 32 >> @@ -88,6 +89,23 @@ struct rte_thash_ctx { >> uint8_t hash_key[0]; >> }; >> >> +uint8_t rte_thash_gfni_supported; > > .. = 0; > ? > This goes against style: ERROR:GLOBAL_INITIALISERS: do not initialise globals to 0 I'll init it inside the RTE_INIT section >> + >> +void >> +rte_thash_complete_matrix(uint64_t *matrixes, uint8_t *rss_key, int size) >> +{ >> + int i, j; >> + uint8_t *m = (uint8_t *)matrixes; >> + >> + for (i = 0; i < size; i++) { >> + for (j = 0; j < 8; j++) { >> + m[i * 8 + j] = (rss_key[i] << j)| >> + (uint8_t)((uint16_t)(rss_key[i + 1]) >> >> + (8 - j)); >> + } >> + } >> +} >> + >> static inline uint32_t >> get_bit_lfsr(struct thash_lfsr *lfsr) >> { >> @@ -759,3 +777,11 @@ rte_thash_adjust_tuple(struct rte_thash_ctx *ctx, >> >> return ret; >> } >> + >> +RTE_INIT(rte_thash_gfni_init) >> +{ >> +#ifdef __GFNI__ >> + if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_GFNI)) >> + rte_thash_gfni_supported = 1; >> +#endif >> +} >> diff --git a/lib/hash/rte_thash.h b/lib/hash/rte_thash.h >> index 76109fc..e3f1fc6 100644 >> --- a/lib/hash/rte_thash.h >> +++ b/lib/hash/rte_thash.h >> @@ -28,6 +28,7 @@ extern "C" { >> #include >> #include >> #include >> +#include >> >> #if defined(RTE_ARCH_X86) || defined(__ARM_NEON) >> #include >> @@ -113,6 +114,8 @@ union rte_thash_tuple { >> }; >> #endif >> >> +extern uint8_t rte_thash_gfni_supported; >> + >> /** >> * Prepare special converted key to use with rte_softrss_be() >> * @param orig >> @@ -223,6 +226,25 @@ rte_softrss_be(uint32_t *input_tuple, uint32_t input_len, >> return ret; >> } >> >> +/** >> + * Converts Toeplitz hash key (RSS key) into matrixes required >> + * for GFNI implementation >> + * >> + * @warning >> + * @b EXPERIMENTAL: this API may change without prior notice. >> + * >> + * @param matrixes >> + * pointer to the memory where matrixes will be writen. >> + * Note: the size of this memory must be equal to size * 8 >> + * @param rss_key >> + * pointer to the Toeplitz hash key >> + * @param size >> + * Size of the rss_key in bytes. >> + */ >> +__rte_experimental >> +void >> +rte_thash_complete_matrix(uint64_t *matrixes, uint8_t *rss_key, int size); >> + >> /** @internal Logarithm of minimum size of the RSS ReTa */ >> #define RTE_THASH_RETA_SZ_MIN 2U >> /** @internal Logarithm of maximum size of the RSS ReTa */ >> diff --git a/lib/hash/rte_thash_gfni.h b/lib/hash/rte_thash_gfni.h >> new file mode 100644 >> index 0000000..8f89d7d >> --- /dev/null >> +++ b/lib/hash/rte_thash_gfni.h >> @@ -0,0 +1,229 @@ >> +/* SPDX-License-Identifier: BSD-3-Clause >> + * Copyright(c) 2021 Intel Corporation >> + */ >> + >> +#ifndef _RTE_THASH_GFNI_H_ >> +#define _RTE_THASH_GFNI_H_ >> + >> +/** >> + * @file >> + * >> + * Optimized Toeplitz hash functions implementation >> + * using Galois Fields New Instructions. >> + */ >> + >> +#include >> + >> +#ifdef __cplusplus >> +extern "C" { >> +#endif >> + >> +#ifdef __GFNI__ >> + >> +#define RTE_THASH_FIRST_ITER_MSK 0x0f0f0f0f0f0e0c08 >> +#define RTE_THASH_PERM_MSK 0x0f0f0f0f0f0f0f0f >> +#define RTE_THASH_FIRST_ITER_MSK_2 0xf0f0f0f0f0e0c080 >> +#define RTE_THASH_PERM_MSK_2 0xf0f0f0f0f0f0f0f0 >> +#define RTE_THASH_REWIND_MSK 0x0000000000113377 >> + >> +__rte_internal >> +static inline void >> +__rte_thash_xor_reduce(__m512i xor_acc, uint32_t *val_1, uint32_t *val_2) >> +{ >> + __m256i tmp_256_1, tmp_256_2; >> + __m128i tmp128_1, tmp128_2; >> + uint64_t tmp_1, tmp_2; >> + >> + tmp_256_1 = _mm512_castsi512_si256(xor_acc); >> + tmp_256_2 = _mm512_extracti32x8_epi32(xor_acc, 1); >> + tmp_256_1 = _mm256_xor_si256(tmp_256_1, tmp_256_2); >> + >> + tmp128_1 = _mm256_castsi256_si128(tmp_256_1); >> + tmp128_2 = _mm256_extracti32x4_epi32(tmp_256_1, 1); >> + tmp128_1 = _mm_xor_si128(tmp128_1, tmp128_2); >> + >> + tmp_1 = _mm_extract_epi64(tmp128_1, 0); >> + tmp_2 = _mm_extract_epi64(tmp128_1, 1); >> + tmp_1 ^= tmp_2; >> + >> + *val_1 = (uint32_t)tmp_1; >> + *val_2 = (uint32_t)(tmp_1 >> 32); >> +} >> + >> +__rte_internal >> +static inline __m512i >> +__rte_thash_gfni(uint64_t *mtrx, uint8_t *tuple, uint8_t *secondary_tuple, >> + int len) > > Here and in other fast-path functions: > const uint64_t *mtrx > Agree >> +{ >> + __m512i permute_idx = _mm512_set_epi8(7, 6, 5, 4, 7, 6, 5, 4, >> + 6, 5, 4, 3, 6, 5, 4, 3, >> + 5, 4, 3, 2, 5, 4, 3, 2, >> + 4, 3, 2, 1, 4, 3, 2, 1, >> + 3, 2, 1, 0, 3, 2, 1, 0, >> + 2, 1, 0, -1, 2, 1, 0, -1, >> + 1, 0, -1, -2, 1, 0, -1, -2, >> + 0, -1, -2, -3, 0, -1, -2, -3); >> + >> + const __m512i rewind_idx = _mm512_set_epi8(0, 0, 0, 0, 0, 0, 0, 0, >> + 0, 0, 0, 0, 0, 0, 0, 0, >> + 0, 0, 0, 0, 0, 0, 0, 0, >> + 0, 0, 0, 0, 0, 0, 0, 0, >> + 0, 0, 0, 0, 0, 0, 0, 0, >> + 0, 0, 0, 59, 0, 0, 0, 59, >> + 0, 0, 59, 58, 0, 0, 59, 58, >> + 0, 59, 58, 57, 0, 59, 58, 57); >> + const __mmask64 rewind_mask = RTE_THASH_REWIND_MSK; >> + const __m512i shift_8 = _mm512_set1_epi8(8); >> + __m512i xor_acc = _mm512_setzero_si512(); >> + __m512i perm_bytes = _mm512_setzero_si512(); >> + __m512i vals, matrixes, tuple_bytes, tuple_bytes_2; >> + __mmask64 load_mask, permute_mask, permute_mask_2; >> + int chunk_len = 0, i = 0; >> + uint8_t mtrx_msk; >> + const int prepend = 3; >> + >> + for (; len > 0; len -= 64, tuple += 64) { > > What will happen if len < 64? > If len < 64 then only necessary number of bytes will be loaded into the ZMM register (see load_mask). After, internal loop with actual calculations will be executed for all loaded bytes. >> + if (i == 8) >> + perm_bytes = _mm512_maskz_permutexvar_epi8(rewind_mask, >> + rewind_idx, perm_bytes); >> + >> + permute_mask = RTE_THASH_FIRST_ITER_MSK; >> + load_mask = (len >= 64) ? UINT64_MAX : ((1ULL << len) - 1); >> + tuple_bytes = _mm512_maskz_loadu_epi8(load_mask, tuple); >> + if (secondary_tuple) { >> + permute_mask_2 = RTE_THASH_FIRST_ITER_MSK_2; >> + tuple_bytes_2 = _mm512_maskz_loadu_epi8(load_mask, >> + secondary_tuple); >> + } >> + >> + chunk_len = __builtin_popcountll(load_mask); >> + for (i = 0; i < ((chunk_len + prepend) / 8); i++, mtrx += 8) { >> + perm_bytes = _mm512_mask_permutexvar_epi8(perm_bytes, >> + permute_mask, permute_idx, tuple_bytes); >> + >> + if (secondary_tuple) >> + perm_bytes = >> + _mm512_mask_permutexvar_epi8(perm_bytes, >> + permute_mask_2, permute_idx, >> + tuple_bytes_2); >> + >> + matrixes = _mm512_maskz_loadu_epi64(UINT8_MAX, mtrx); >> + vals = _mm512_gf2p8affine_epi64_epi8(perm_bytes, >> + matrixes, 0); >> + >> + xor_acc = _mm512_xor_si512(xor_acc, vals); >> + permute_idx = _mm512_add_epi8(permute_idx, shift_8); >> + permute_mask = RTE_THASH_PERM_MSK; >> + if (secondary_tuple) >> + permute_mask_2 = RTE_THASH_PERM_MSK_2; >> + } >> + } >> + >> + int rest_len = (chunk_len + prepend) % 8; >> + if (rest_len != 0) { >> + mtrx_msk = (1 << (rest_len % 8)) - 1; >> + matrixes = _mm512_maskz_loadu_epi64(mtrx_msk, mtrx); >> + if (i == 8) { >> + perm_bytes = _mm512_maskz_permutexvar_epi8(rewind_mask, >> + rewind_idx, perm_bytes); >> + } else { >> + perm_bytes = _mm512_mask_permutexvar_epi8(perm_bytes, >> + permute_mask, permute_idx, tuple_bytes); >> + >> + if (secondary_tuple) >> + perm_bytes = >> + _mm512_mask_permutexvar_epi8( >> + perm_bytes, permute_mask_2, >> + permute_idx, tuple_bytes_2); >> + } >> + >> + vals = _mm512_gf2p8affine_epi64_epi8(perm_bytes, matrixes, 0); >> + xor_acc = _mm512_xor_si512(xor_acc, vals); >> + } >> + >> + return xor_acc; >> +} >> + >> +/** >> + * Calculate Toeplitz hash. >> + * >> + * @warning >> + * @b EXPERIMENTAL: this API may change without prior notice. >> + * >> + * @param m >> + * Pointer to the matrices generated from the corresponding >> + * RSS hash key using rte_thash_complete_matrix(). >> + * @param tuple >> + * Pointer to the data to be hashed. Data must be in network byte order. >> + * @param len >> + * Length of the data to be hashed. >> + * @return >> + * Calculated Toeplitz hash value. >> + */ >> +__rte_experimental >> +static inline uint32_t >> +rte_thash_gfni(uint64_t *m, uint8_t *tuple, int len) >> +{ >> + uint32_t val, val_zero; >> + >> + __m512i xor_acc = __rte_thash_gfni(m, tuple, NULL, len); >> + __rte_thash_xor_reduce(xor_acc, &val, &val_zero); >> + >> + return val; >> +} >> + >> +/** >> + * Calculate Toeplitz hash for two independent data buffers. >> + * >> + * @warning >> + * @b EXPERIMENTAL: this API may change without prior notice. >> + * >> + * @param m >> + * Pointer to the matrices generated from the corresponding >> + * RSS hash key using rte_thash_complete_matrix(). >> + * @param tuple_1 >> + * Pointer to the data to be hashed. Data must be in network byte order. >> + * @param tuple_2 >> + * Pointer to the data to be hashed. Data must be in network byte order. >> + * @param len >> + * Length of the largest data buffer to be hashed. >> + * @param val_1 >> + * Pointer to uint32_t where to put calculated Toeplitz hash value for >> + * the first tuple. >> + * @param val_2 >> + * Pointer to uint32_t where to put calculated Toeplitz hash value for >> + * the second tuple. >> + */ >> +__rte_experimental >> +static inline void >> +rte_thash_gfni_x2(uint64_t *mtrx, uint8_t *tuple_1, uint8_t *tuple_2, int len, >> + uint32_t *val_1, uint32_t *val_2) > > Why just two? > Why not uint8_t *tuple[] > ? > x2 version was added because there was unused space inside the ZMM which holds input key (input tuple) bytes for a second input key, so it helps to improve performance in some cases. Bulk version wasn't added because for the vast majority of cases it will be used with a single input key. Hiding this function inside .c will greatly affect performance, because it takes just a few cycles to calculate the hash for the most popular key sizes. >> +{ >> + __m512i xor_acc = __rte_thash_gfni(mtrx, tuple_1, tuple_2, len); >> + __rte_thash_xor_reduce(xor_acc, val_1, val_2); >> +} >> + >> +#else /* __GFNI__ */ >> + >> +static inline uint32_t >> +rte_thash_gfni(uint64_t *mtrx __rte_unused, uint8_t *key __rte_unused, >> + int len __rte_unused) >> +{ >> + return 0; >> +} >> + >> +static inline void >> +rte_thash_gfni_x2(uint64_t *mtrx __rte_unused, uint8_t *tuple_1 __rte_unused, >> + uint8_t *tuple_2 __rte_unused, int len __rte_unused, >> + uint32_t *val_1 __rte_unused, uint32_t *val_2 __rte_unused) >> +{ >> + > > That seems inconsistent with dummy rte_thash_gfni() above. > Should be: > *val_1 = 0; *val_2 = 0; > I think. > Agree >> +} >> + >> +#endif >> + >> +#ifdef __cplusplus >> +} >> +#endif >> + >> +#endif /* _RTE_THASH_GFNI_H_ */ >> diff --git a/lib/hash/version.map b/lib/hash/version.map >> index ce4309a..cecf922 100644 >> --- a/lib/hash/version.map >> +++ b/lib/hash/version.map >> @@ -39,10 +39,12 @@ EXPERIMENTAL { >> rte_hash_rcu_qsbr_add; >> rte_thash_add_helper; >> rte_thash_adjust_tuple; >> + rte_thash_complete_matrix; >> rte_thash_find_existing; >> rte_thash_free_ctx; >> rte_thash_get_complement; >> rte_thash_get_helper; >> rte_thash_get_key; >> + rte_thash_gfni_supported; >> rte_thash_init_ctx; >> }; >> -- >> 2.7.4 > -- Regards, Vladimir