From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 15CA342D72 for ; Tue, 27 Jun 2023 15:08:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 03E3540F18; Tue, 27 Jun 2023 15:08:46 +0200 (CEST) Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on2052.outbound.protection.outlook.com [40.107.15.52]) by mails.dpdk.org (Postfix) with ESMTP id 6967440A7F for ; Sun, 25 Jun 2023 01:09:11 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Sx4L0VIoxuyzd3BDNLes8Xd20uVTA8fyGoD0BMwk6MI7C6k4qSawuOY0wyhKnfeW1vbqSkihU20xru5IDA8yOYcAiNtMs9dXut0yg22CzcMQtJtM8Y4Y02Jmn1WsTty3sn6QeIqDAvVd9Ck5X5toqk9vr5t2tTbyOGDzco4F3SRz7pLXCB2Ye7xZ4SjUy0wLN2T06ygW26UgYQY/Pg7bo84/y+PL2PBXV8R8mLSQBq0rYMN7kkbJOD0kWPQvzlkIp1RQhCx0uX+BmP3oJQGXIBbJiCo7Vr7PNvcueOhi9cw9kThhA+NzjyrAaXe5EURRhu3ZvcUD0TsHK28WLgjjEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8gQG73N3LgZaRPW9syNJly1cNK4E9dcBSkLncYpOH3w=; b=V5lFQtCjnQQ3bfzhkTrvX105ntNbVr2ValUpYcV6a3WJE6o+mcoe0sclMB1jcvrwfitjKqQRHIeYHpVw2sBYDWNt7f7X0/xVJ1b0HgcAChUSGCN+/Rh+3iRGGNgJ1NC+E7qtKJfQ/TxUkVKPXSUxfFEICMSDCHcxwVNS9C2MS9DKaJFK79ScjMbeVVQS6ntfyMRG2G5/ZC/vbH7NwNda8m/tR5EvSp5/w9bECaH87xuKA5soE9AkNFiy5EHF6hp74nFy474B1chr9/9p0KITSI48kd/A3aSIqaNr2snLqfpYHKsAxj4FNB16stG36IQEdJ41dLX3OufMFkAzq/3uiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=qmul.ac.uk; dmarc=pass action=none header.from=qmul.ac.uk; dkim=pass header.d=qmul.ac.uk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qmul.ac.uk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8gQG73N3LgZaRPW9syNJly1cNK4E9dcBSkLncYpOH3w=; b=lZi7nmdybksqieGGc26c/6p/751SuZZYmTHsYkpHNw4ZGXpVxSoomuef5OpZ4ABt0i9j4aFPZ/TXg/YjJm04vNVyWJZbbUQL77A0kK4Nmn6lGV2s8KHOtfTlh/g9O9VYvLIbmU6gPXaNqLVj2tBSzNz03iO6kgoYQuHw3j9rJA0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=qmul.ac.uk; Received: from AM0PR07MB6402.eurprd07.prod.outlook.com (2603:10a6:20b:15e::16) by DBAPR07MB6855.eurprd07.prod.outlook.com (2603:10a6:10:19b::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6521.26; Sat, 24 Jun 2023 23:09:09 +0000 Received: from AM0PR07MB6402.eurprd07.prod.outlook.com ([fe80::879e:39a3:3985:dea0]) by AM0PR07MB6402.eurprd07.prod.outlook.com ([fe80::879e:39a3:3985:dea0%5]) with mapi id 15.20.6521.026; Sat, 24 Jun 2023 23:09:09 +0000 Message-ID: <4f7143b7-7a94-a082-98be-0d5c403018ef@qmul.ac.uk> Date: Sun, 25 Jun 2023 00:09:07 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Content-Language: en-US To: users@dpdk.org From: Alireza Sanaee Subject: High latency on Mellanox Bluefield II from PCI-e crossing Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO3P265CA0011.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:bb::16) To AM0PR07MB6402.eurprd07.prod.outlook.com (2603:10a6:20b:15e::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM0PR07MB6402:EE_|DBAPR07MB6855:EE_ X-MS-Office365-Filtering-Correlation-Id: 2d864b11-ac34-43a8-7486-08db750803b0 X-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: siRarlkzJLrdrdMuVNigsoSQZsYC0O8Ixw8fM7jMOXgsJuB1MlMteFTaMTT3CftBfaz2bIwl9mIO35tXXt+TUlIimQsMZfz8pUSIpvzBsDLZPfqSvdbwyBv/keTQZcEFZgUUQr5qT+/dZgd+/7l/so5F1hNg85Isxu5rjOwSmU5CRBCRpYAZcf2LsfvStN/nY6+u15jn+3DtC7z7lighbKSB+Wt2z+Q25CFTTT73XGJQUrCRuxoZNZC90fVBkgVymJI6d1ascpkSwa1/EBMTHUUK4nhNKGXS0c8z2VTdfoSp52cIIawFdGKJ7UpU3joscgh+gW6ci86CsMN2rVTMb7ZkCvXzWUwAFPq8whN2llp8Cd/cJ6WKkTO8SQX9x0MVpYDQYFa7MI+bc73wNxDPG26iDL3L8GM6gp7WAQhyFxr4sVVr0mTFuz7uTqGDPCSBspgTe1ppwcdYZHWsFnMnT9DXuzzedMAG3j7ITNvj08vL75m3uoc5RjO62mnYx4XWciwT2ACH+aKpSRc0PEmOoANTB9ls2idPgijans23dLpaVQ31RrIGC80ioD6zWcQ3gn/zdwb6FyUYah/K3QPzvMUqkzRWtIKFfE7cCeKS8mvexIh61rr8JVVX1dfQ9hA1CpZ29MTvy5PfRs3cnNnRGw6qlSPC9/UQgObKYX2fki/mQhwmWgYKza3EnSLVM0Im X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR07MB6402.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(39850400004)(346002)(376002)(396003)(136003)(366004)(451199021)(478600001)(83380400001)(2616005)(6512007)(2906002)(6506007)(186003)(4744005)(6486002)(5660300002)(36756003)(38100700002)(66946007)(316002)(786003)(31696002)(8936002)(66556008)(8676002)(66476007)(41300700001)(6916009)(86362001)(31686004)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SjhQaGxDb21KYjdhd2xNUkdkQ3JUVWxQMERiNjdEQWtPMEluYXBZS3RnaXFm?= =?utf-8?B?eG5UcUlQL0pvMVVYSDhYQ1ZFK1doNHkwYWgwUHk4ZFhRbERWM1dUREhUQ2hi?= =?utf-8?B?ZndueXFkTlBBWjhSQkxlT3JoV0xWY0tMRlZDdHc5Y0trQU54UWFhZ0JXcnYv?= =?utf-8?B?NUlSVWtjdzREQWYyeDVyVCtNbkxZbHZhY2FnUGRGSG10bzBQWWF6SXp1amFW?= =?utf-8?B?T0tHak5mRXZka0p5a2U3R3cxaGlwUU1mckdJWGM5d0NWUFpCMVk1RW9oNWhJ?= =?utf-8?B?bGlFS0ZIQXZCOXE4bmszSEMwQWptOE9CNWpVV0NUdXd4NG9sTVRLNlNyZDd6?= =?utf-8?B?M1hIa28rVXVJYk16a2hOaVZUR0dEWTc5cjJCRG5HR2k4c3lENUpEaEM1N1dk?= =?utf-8?B?M0JmUURSQys5cFFTY2ptcyswVU9VM3B0a055STNVcVFVUzlqMnFiNU03bVlJ?= =?utf-8?B?Zy9Ga2N0T1UvNHN0cVRhdWsyNEpMOHE1T1g4RHk0SHU4YStjdG1lUEpVMXBv?= =?utf-8?B?aENHMEJJajF1T2ZQalA4Y0FCYVA2WVNmQVYzZEhBSkNNOWxWOVJWbkJvZGll?= =?utf-8?B?cWpiZlU5bjI0NUgrSXhuaXZhUVhVd1hiZTVLNGdUVlhqVXFBbUdETlYvSllL?= =?utf-8?B?NmR0cE8zTUl5TjN2aXBYV2Vuc3VHaWZ0bVRtY0N0Q2s0QWNFeFdBUHlqajc0?= =?utf-8?B?QnV3VUphSmR4aHFRVytTMTFrV3ZCQTc1UXZJZy9lVm1FVkxQM093cUFJSExv?= =?utf-8?B?L3J1NzlGaE5kVGFQSVhibC9sU0hkSlFWRWJ5K3NYRWdIR2x6MmtxbWRyc3J3?= =?utf-8?B?Y29STEw0UWZnVjFoc0hYeHRBbWVnWmdvSDY1Z2VTUkpueDdteGIyY1VZZjFm?= =?utf-8?B?Zi83K2kyY2QyUUpobzMwRVR1eE5LR0oxbS9YZVMzWnl3SVJmUy9tVmo3STB1?= =?utf-8?B?b2ZFaE13UTJmc1dWczcvNVZ4SVZ3RUpqVlJyWmRvS1RvMlBzQTRTS1Uxcndk?= =?utf-8?B?NHFTRWxUMk9Od1hhY3BsU29TbEhhS09tZmx1WnlNekRBWi9TZndoMkVlbVJC?= =?utf-8?B?a1JRUEN6WlFOWDlBMkJXb29HT0VpbVM4bXVHa1N5bVo3NmFIMnQxUGJsdUZV?= =?utf-8?B?cUpVVUlWZlhWbUNabnp2TmdqamRoMmhVZlpEdmNSaWRMUDUzZVRSZEtITG5v?= =?utf-8?B?SjVlV1dMR3cxTCtBMTJtVnIxdkNsRHRLaHowUjZ3NEI1MG9UVXp0VFp3b3Nn?= =?utf-8?B?aXEvbDRpd0k3dzVkSFBwUzd1TmxwZmwwVkJ5UVNpV01zTTdlMjU5aE5McnZO?= =?utf-8?B?WUFSUGw4L1JPVDIwbHQxa1Q4QUpJZ1hMQkNzQlJEMGJIeFhSSjBxQW53OUM1?= =?utf-8?B?NHpPbDlTTDJjVDN0d05selY1NldiWGVEWEFYaVRjbkRBYytwd3RtQm1RSURQ?= =?utf-8?B?cVo4cTlKRWpkZkZ0VVZSZURET2RlVHQ3N0x4a3hQeUdzekZVQXJJanVraEhK?= =?utf-8?B?SVFNSTNkdUVISjhLdlN1L3ZoU1pYNjdpNG9kRFpUc1F6OEVyRjJwUnEyZTRh?= =?utf-8?B?K21mQnpBaGVzMEprdmIxQXllTTNMSVFrWWRlU3k0T1V1aE9NVWpFS3BOZUlC?= =?utf-8?B?b2RDa0M5NDg3K1YzUHBqYXMxTTJHTkwvaHJIejIxa2lsa3llSTJFTDhlYVNv?= =?utf-8?B?eURoZ2FoMUwxTnUweXM2L05wdG9BeEpRaHZ2emVPUnZTK0VRRnhlYUgrUExM?= =?utf-8?B?ZjdQMmFDUXlBZUhkSHZYTVF4TThhL09raU5qZEpmeWtja0xpTnhxL3FhT3ZK?= =?utf-8?B?MDdOV25oMWlTTFlwVGVzUG5FOElaL2dvYjg4c2IxTm02WGhHS0lpV1RqU3pP?= =?utf-8?B?S1FkSkltQTZyZSsyY0s3a2Q3VXhraDJ3MHV2RFFBY1dEeUVDekFGc2hRNGlF?= =?utf-8?B?TlJURlUwTWljMWsvdndHS1JsdkQvVDhDSklEVmhLWTM4UVp0RWZ1TGhuTDd3?= =?utf-8?B?RldVemtTeExDRGNLQ2hDWWZ2aXFEYnRvSGdBSDM5alhMaS9pT1NCelgrcHFO?= =?utf-8?B?UDZCR2VZM256T1NRcURZdjEzcVdFcDE1R1o0aldLTFczcWtOZlN0dVg1RVM0?= =?utf-8?B?dG1WY0V3YW1oWnhsRlE5Y0VjTTNBRkdpRTdxTjhWbnVnVFJWdVFUWUw5L1B5?= =?utf-8?Q?sq7dSiSKBdjEoJrGgXGuit59aTPON+nufFxsCgt1/ZUB?= X-OriginatorOrg: qmul.ac.uk X-MS-Exchange-CrossTenant-Network-Message-Id: 2d864b11-ac34-43a8-7486-08db750803b0 X-MS-Exchange-CrossTenant-AuthSource: AM0PR07MB6402.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2023 23:09:08.7586 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 569df091-b013-40e3-86ee-bd9cb9e25814 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ukx+vW7P2925hq59uGy/w3Mk186UTqa1mPe1yv5MtcHrz9D6oNj8ZO9oHychL3MyvO5VJQ/qm+Z5NL+v612Ctg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR07MB6855 X-Mailman-Approved-At: Tue, 27 Jun 2023 15:08:45 +0200 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi everyone, I wrote a simple loop back application and measured latency at low load around 1Kpps.  I am experiencing severe extra latency at median around 2-4us when crossing PCI-e. This is not the case for connectx5 cards. I am wondering if anybody else noticed this issue or not, and if there is fix. Thanks, Alireza