From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D53342B8F; Wed, 24 May 2023 18:36:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1549540156; Wed, 24 May 2023 18:36:46 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 672CF400EF for ; Wed, 24 May 2023 18:36:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1684946203; x=1716482203; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=MdC5Q1r39uVb7vm/7SyRWuMHIELsi/3QDn65JjzLxYM=; b=gHO6oAnLFYvBsFCMi+7/Famfl/uTNZNEl246+8iKKSkRaqvphGN1u4NX aRM3Vuqk4j5PImfO5bUXZZSbfb5MQ62JTphrTz8UQU561X26DPWqHTbqS vHDJaG7QOTvSZzb6ErhC8EcCrYBhRyWa8OjTgtDbbkTHbZ6MNE1dfdPG4 kxcWSrRE7wGdOrZksbt9vLzzfm/mBJ4YIwaWPa8uxjKo8Mmc1gfrATu2M FjHvHB3w53N6sj4wnQn7UWnhzgnUTE3746gfgEGbLd1xaYJcw7TQZ/EEP qEh/p8UlWfLFz2FiSoCy2aBNkXzTRcm7+8d6BWnowpTM7dAx+exhjngas A==; X-IronPort-AV: E=McAfee;i="6600,9927,10720"; a="356849846" X-IronPort-AV: E=Sophos;i="6.00,189,1681196400"; d="scan'208";a="356849846" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 May 2023 09:36:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10720"; a="704421180" X-IronPort-AV: E=Sophos;i="6.00,189,1681196400"; d="scan'208";a="704421180" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by orsmga002.jf.intel.com with ESMTP; 24 May 2023 09:36:41 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 24 May 2023 09:36:41 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 24 May 2023 09:36:41 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Wed, 24 May 2023 09:36:41 -0700 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.173) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Wed, 24 May 2023 09:36:41 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XRcYFLvn98/F46LZLsR9JdYf+AChyA5LyNE/XnqOpnZkrxN0m5GiLq95hNJIQUzB7ieef1ECXOq7+UrfcgD9U2mWlygV2S9pxWGLXD9xurb2h8xHPhl4ql7v2+aj4AfsY0Vl2NcaQVg1zcqaa74HAIRwlnUFo+13r02bom+njW4OAEXe0DNJMMJTBF6JSxCOu7itxVVKc/3npVIdnKVhaBDK8UiI4stloH836fLHCtcpoLJQLhkAef+5nIzDlL6ylbZwHWh9t/L531Bxt5WMXrv3t0pO6fedklF1mTYaUyk01t5gKAVolBrFnHL3r51RWiHfsiCfvUWN/WaZCQrgBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ghgJGcmb+/W7jnAR46qJNdcjNiosA+ZNG5Qog5eaF7k=; b=i56MpEEuLOKSx/WVLnOHhC7U6RgUMqaSUSxJG8ShWiW5X9ddEu4Mraep2baqOu3rzoHSVhgi9j1A0U8qM5mi0K7C/Ch4l/kZok0cNuHQslQGleNvD+laUGk+cBtkM4Q+RROFJhU97N3uBftf8gGplQbihxfw1Fh9NgFbMFVtPkq6bH+mxlAL89DnDywaOJbSHYh8wDWHRwnYal7yUDb6/7T9w+LqDZoGYEzCgKoD66iD9UPax3w0suaW7tySq7zcY12Lq7ABqqaAlSCIZmmaGYZklYznaL0wNnVT/IcrNcw1ZPKPJbHPdS1V3jMk5yBImNgkA8ZxQ5uO+iqceh/swA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by CY5PR11MB6162.namprd11.prod.outlook.com (2603:10b6:930:29::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Wed, 24 May 2023 16:36:39 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::b8f3:958:d2c5:2232]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::b8f3:958:d2c5:2232%7]) with mapi id 15.20.6411.028; Wed, 24 May 2023 16:36:38 +0000 Date: Wed, 24 May 2023 17:36:32 +0100 From: Bruce Richardson To: Alireza Sanaee CC: Subject: Re: AF_XDP performance Message-ID: References: <0fba676d-a49b-76fe-6714-0700d5b61749@qmul.ac.uk> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <0fba676d-a49b-76fe-6714-0700d5b61749@qmul.ac.uk> X-ClientProxiedBy: LO4P265CA0131.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:2c6::12) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|CY5PR11MB6162:EE_ X-MS-Office365-Filtering-Correlation-Id: fc4ca909-144e-4a4c-be0f-08db5c750bdb X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aXNgY3F+YNBY7bCTbKskOasoNOr+2ShU1vMWXOgssA2JXFkgaQIefNPOy1MdHfEVewaT++2UnoEkbpZ+K+M9ELs9X4VItnVkaNaqbZp/9xF3DRFYxeN7AcB5ZgKjV34L7Da99MgDPQ2+cnG8fbFzGsa0DoMPw+wUWzzsQUIo0OjGqpNSzu263Bp6cMBNW05Fab3LoGlhssN1BTl7Np4ijSfukDQbp4f/hIZnbsBaHiTGVcDxah3z1PBM0rleDliX41+24NL/tnagarCJ2IVSzOZjxnJrWPf9l3+ItOHD2S99ansv2gHd3pVghd5qdtQgo6dZeJnlBmQN3St25RuQOrod81Y9JCM/swHrQtyDcq3VqxLGsKNaVvHvnOSZDxX1WZ8REho6+N8ZZWmfCHMswdxVr8m3AEOLHnyOVRXDAA63n6Q9MNO1OSbGX8YH3+SDJfVTWWsGZQlwuOfLy0ikIIIu6P5byFmz6ABuo7vRzlADu5t+UDcjAKV4+siT0q/s4jr5bG/deb2GE4tGhrReaXTUUq+3FFI1nWyLKWAO5/i66gWmmDY1Zqv/jb4wYxOqaGLluVsWlrREqOpG4U7grPkuegC0bjDnEB2NxEQ8RYU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(376002)(346002)(136003)(366004)(396003)(39860400002)(451199021)(26005)(38100700002)(6512007)(186003)(6506007)(44832011)(82960400001)(2906002)(83380400001)(41300700001)(6916009)(6486002)(296002)(316002)(6666004)(966005)(478600001)(66556008)(66946007)(66476007)(4326008)(86362001)(8936002)(5660300002)(8676002)(7116003)(15398625002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pACR0RisggniY5BOD0ZVkUO2jZKEY48P/EZt+0EFqiUaP/eqdo5IXHi1vQU5?= =?us-ascii?Q?uOB6GVCnva5KWgb9EADCZa7CdAgsGaHkOm0tZY4bIbsQk0STqlN1YXGKIKIw?= =?us-ascii?Q?fpVpAeud2OUe9VhL3uHO1Uk+AZ5BeF+WegA++nDteJV/bulJTd9odXg+EXky?= =?us-ascii?Q?p1f/CMB47NNOdlNT0dxrCwasxOeYxBf5oi0OinxM4fNKYsKz9E6VS+CqDGm+?= =?us-ascii?Q?t7CQHe1jd0e9uFySVnWZPyi8xaXd5yXU03dEZQ2OwkPIa6C8BLy5+xK7PtZT?= =?us-ascii?Q?Sl2BcGopCbEuFMbwZyWQj/+OoVyKfD6qpF8J14ByNAx+G08HOHHRmAsJpDwc?= =?us-ascii?Q?Chu73HmTq8jBgYnVXA1cEw3kWGWPn1X/kDxkDKLYrXuB5+89Tl0SGiRSvYWh?= =?us-ascii?Q?MqFx93go4OChF4qX/ZP+Gor23S1Wgl38Xv2JmxkbF34Ih6mjZXAqTdPz7Boi?= =?us-ascii?Q?oPnrwiU53ADca+J9hT1bOx/bmc5Ueno87sDAAT/i9rju4Hp+lp4tJgxcOHuS?= =?us-ascii?Q?IcrhjhbxSVNoMSF1mGjSQnF4N4vEmGkSa5mLlTnT629/8oRukwmW4sCZ/vxp?= =?us-ascii?Q?ZP1/VQ+LqlYrl1b9QNh2WDftzALxUMkS8aIRtIq6tlRCH+A8+GujtXww/sf5?= =?us-ascii?Q?mxtmeUHc88lW8geiaQN1LCLLbbVICcrtBqA8hbBAWVqZ5W+eAqzPYogBzovX?= =?us-ascii?Q?gYCpqZuu/UzfQsPTa2M5eM/5mVS9iQ94dt9oLKa3MvqYQYUi6FhiBxQmc7nu?= =?us-ascii?Q?DfIilc4/R3NhKQGqfqkK1bfBDF+7sdrD2PrgK/3j1OHEmjNqjMlJAGytU9ya?= =?us-ascii?Q?+jCNvl2bhxEAtN7oSM3fBxosshf6TOErtaDQda0fqATv+6G+bJ1vWER8mE98?= =?us-ascii?Q?DDPpZYHjif80AFHU4GVEJ0/DXifgHEpB9YsbYGAQnFNGAvvupn6mTY/cPynk?= =?us-ascii?Q?/yqtegk4IanelbjhGgKFYHj5c44Bx5ltWosZ0WZ9wMVVNb7eJOuxfZn+zwwz?= =?us-ascii?Q?Uu+iWgwEuP/obvtdEBbWG0hepacCi/DyiHWmT2ul4FmR9Rb8QfJbJoYvZipb?= =?us-ascii?Q?JCNBOI4lTSLYxdqmX/hpKSlLhcC4fj+zOkZtGL56cPqNE+MkA5Lw9tbQKFTL?= =?us-ascii?Q?2tofPA0Qbl2ece0WKHbWpJrF8rXG2990ffjsIbInn7XQ+wmOo9Ren6LErV3A?= =?us-ascii?Q?9Nl3+Irw5DwW+OEhvRSf/m0/y6HDPYFFWUJwzblBeRWKu3PhgpyJjlX6q9Xl?= =?us-ascii?Q?Y6pdcqtHg8TA6EOxRHCBQLINFcOnXShsK0bzsi/EELAZjQytaQx6LHur0l/e?= =?us-ascii?Q?kecHfq6ief1xgDk5nT73BED40X6MpzY5aFqZ8ORzp+nR+RV23Wa0BMRBJo4T?= =?us-ascii?Q?CzluWqGzOtTh+gLbdjgthHr8E7GjznYyW+ac/zuVVSfHJjO8mOOJmmlsLQLm?= =?us-ascii?Q?3TzZLGzhJGhr/tp2vv5EXAdfOtmWFQpSXlQSXM/5qQhcJvz+MSZZK5IUpIOP?= =?us-ascii?Q?WQsPI4TEMcdfzttcsT4lYmQQRY4IZt/5Nl6RsvjVHGSgWuwtUPWMvX9rN7e+?= =?us-ascii?Q?b+XEGDgigfd7DU9LGHwJMe++xwKV6sCXJ9rhpVxdiwVhyt+oKPTk7tmeg5WB?= =?us-ascii?Q?eg=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: fc4ca909-144e-4a4c-be0f-08db5c750bdb X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2023 16:36:38.5999 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 31B6r9s9Shm/CXLoNabWpNa/aWnODV+m72oESh6WjRchkgp9D4UA3UIFTyQ29LYTacL3K0AaACY1GgEgyVZk9cjlguhJw6IadDGHYtVD5dc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6162 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, May 24, 2023 at 01:32:17PM +0100, Alireza Sanaee wrote: > Hi everyone, > > I was looking at this deck of slides > https://www.dpdk.org/wp-content/uploads/sites/35/2020/11/XDP_ZC_PMD-1.pdf > > I tried to reproduce the results with the testpmd application. I am > working with BlueField 2 NIC and I could sustain ~10Mpps with testpmd > with AF_XDP, and about 20Mpps without AF_XDP on the RX drop experiment. I > was wondering why AF_XDP so lower compared to PCI-e scenario given the > fact that both cases are zero-cpy. Is it because of the frame size? > While I can't claim to explain all the differences, in short I believe the AF_XDP version is just doing more work. With a native DPDK driver, the driver takes the packet descriptors directly from the NIC RX ring and uses the metadata to construct a packet mbuf, which is returned to the application. With AF_XDP, however, the NIC descriptor ring is not directly accessible by the app. Therefore the processing is (AFAIK): * kernel reads NIC descriptor ring and processes descriptor * kernel calls BPF program for the received packets to determine what action to take, e.g. forward to socket * kernel writes an AF_XDP descriptor to the AF_XDP socket RX ring * application reads the AF_XDP ring entry written by the kernel and then creates a DPDK mbuf to return to the application. There are also other considerations around potential cache locality of descriptors too that could affect things, but I would expect the extra descriptor processing work outlined above probably explains most of the difference. Regards, /Bruce