From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E440E42AA0; Tue, 9 May 2023 08:54:51 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BDAB5410D7; Tue, 9 May 2023 08:54:51 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 9D046400EF for ; Tue, 9 May 2023 08:54:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1683615289; x=1715151289; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=KCeegyB6EAJwpx+yeL5vxQveaY5ZZbSVfIiBt9Y2cwk=; b=AyjlerkPInoErfQCYSePvdOrS9S3GJBwy4+728XaG1RTAodGvJCVR00u rUsN/MMgbTZLBWC7mKVMAe9/+xMjUSb13QaeX3ZpNkm+69GVkKZjwCKeP AR2tBQvfnBfdBQN7+wCQi9VRolHWya6flNatLxZWQQDxIiIbnpwZdJkZ/ vKqOYKU1aWtnbqQQsDMRJUjZOnioiPf7nfKeAIDUNcFc+dce2YkJu+CwF kOlfC8UD6oAZPfkx7Hqz1GpEmRfGVvjaiI0fBss8IIBoI9uPP7IWJjQMX 5qdRyhhEin1GrKldkjQ0xQ3JvGbmxi9LkNvDxhiBq8SuZI+6KuNR7BLRm g==; X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="413090910" X-IronPort-AV: E=Sophos;i="5.99,261,1677571200"; d="scan'208";a="413090910" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 May 2023 23:54:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10704"; a="768335582" X-IronPort-AV: E=Sophos;i="5.99,261,1677571200"; d="scan'208";a="768335582" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by fmsmga004.fm.intel.com with ESMTP; 08 May 2023 23:54:48 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Mon, 8 May 2023 23:54:47 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Mon, 8 May 2023 23:54:47 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.171) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Mon, 8 May 2023 23:54:47 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Uy8jhdOyO+qBRgl3KrNWol18xBbf3uIhwZmF1yNE/JZYKSZsx+eaITjd/35kfXt8IW2mXgXr/i4oIQYS1148wx5kSP40y//SF73ciMdFcag2hhoJ/hQT3qGwSVxXsxYIwBW539Z6u67C2ZPYs6mVFs1gyamvspz/w0UQvwKZ6oUEokvut61+fGKlbdQzqW0TJhzId5L+Pa+2ZHvulH+EyHFtSIala7lYHMSAy+R5yKXDQF9mIsTpedNs69mqjlbwDUvWsKPkedAK+ScTHT4G8YvECd84dsxGCTkGOtWEB46pF89nN5KxF9sYFm/OxCZF3udEnaLQwEQTr7sf7sbXQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FnEG1IPsujYEXLkcjIu7rSSqDC2uAIL8YPMRz/Xvd6M=; b=oYQKCzJ5TffMLGeT+dQ/wZS02vsc2jEuT0sffFdy7ra8l+OjZhfaxCe/z2yJebI1j2OR1O+p0UWrrg2UzXvv9BLR0DNdbxrkxUJTAL87JnGprACZv0vf5c+LtP0KVQDkrXDYSem0JiUDtlIf+h+XmEsxHNDqqAOYzew3PZPQj4T4mivEvJ4WkRrjrZgzkzlGxdMI2RyBVlnQrqQoXGEICvkgIzhxvREGhhFKuuUKEVRrbSpOWb5ovWDQ/DBEfEmQV2k3qA0sHhKaUMk2EqmenmAEyS3PjMkqYPwSpzkhgBMUvxpdm1QPBfTUuGmMGd2MzBf1PP0Q2rbL23582+7gGg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by PH8PR11MB6832.namprd11.prod.outlook.com (2603:10b6:510:22c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.33; Tue, 9 May 2023 06:54:44 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::5797:849:7a10:560d]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::5797:849:7a10:560d%6]) with mapi id 15.20.6363.032; Tue, 9 May 2023 06:54:44 +0000 From: "Xia, Chenbo" To: Nipun Gupta , "dev@dpdk.org" , "thomas@monjalon.net" , "david.marchand@redhat.com" CC: "ferruh.yigit@amd.com" , "harpreet.anand@amd.com" , "nikhil.agarwal@amd.com" Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus Thread-Topic: [PATCH v4 1/4] bus/cdx: introduce cdx bus Thread-Index: AQHZgZ7osKEL5WeqgUyEI4RSX18KwK9Rgb8A Date: Tue, 9 May 2023 06:54:44 +0000 Message-ID: References: <20230124140746.594066-1-nipun.gupta@amd.com> <20230508111812.2655-1-nipun.gupta@amd.com> <20230508111812.2655-2-nipun.gupta@amd.com> In-Reply-To: <20230508111812.2655-2-nipun.gupta@amd.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SN6PR11MB3504:EE_|PH8PR11MB6832:EE_ x-ms-office365-filtering-correlation-id: e7ad9b4d-fb50-4636-7186-08db505a4598 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: onVYYgvJO1MLWzcUOJ/PnB2x/gFYuedyiD79+GaGIjKigCfSO0j/ZdKggjz4laXLPZ+CIb7gz+08dyGDOQRuSodFM/jt7Hsiq3n4th3SVnEVfVV1NQrCgD0FlP9dYNHigmzA7sJ5NQ74hrevTzbvJfpKTXtSILZmPFXi2cuf4zLR2UZeCM0xC2roTYc1eIh3L15hWERXvcgkznk8OoZEyD5j45KRv8WvXW0LTKpmKodvQRMBIWVbB/ygQy0lZA/hHyLPJXMwkTBcpEmjDopvWB8czxKhG4KLHEeckV1maprz34eeOEjzJ4ap8CNeb47jLHXoyixY9ECyfZczXxNJeii6lcrVEOmXsej6tCbaD1vyeN5KLipH7EqesQpEPOkLJcQ9lWPLUYQl2IuCC7uGUbUiBpLeUB2Fg1xrBCI1cN4cani3lhnqxQ5fyPWxu8uWAIO4r0YcSqkS61DKn2SJdH5xDN/Z079KvkoR3MIQGI/iIHKSL0CMHuyMJMBX/n9OW8XK4fYEvKLkbQdcsZcRszsd+uTbK7f8Jki/QdUyKQUQtov9zBWdkc6FIaFvXnhm3Wh8PX84nxkAc4jjQng7UYRkzd7lD5I0kcR0xy14XgnNhfLd8bQ5pKaDwsT5RIrRNoahCrtwR/NYopQpD0l0NQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(39860400002)(366004)(396003)(376002)(346002)(136003)(451199021)(33656002)(86362001)(54906003)(110136005)(316002)(76116006)(66946007)(66556008)(66476007)(66446008)(4326008)(64756008)(7696005)(478600001)(71200400001)(55016003)(8676002)(8936002)(5660300002)(52536014)(2906002)(41300700001)(38070700005)(82960400001)(38100700002)(122000001)(186003)(26005)(53546011)(9686003)(6506007)(83380400001)(21314003); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RxxdBEh9IxP4ZQfg+YNLy1NXe+KHHe39O0uN0NKFzF5XiHNQDFYj5EAywhdT?= =?us-ascii?Q?9TkBENcRUN0lo2EkvHjsRGK1a6pHhvDWo1Lcyj4sDQmYM88b8HKW++XKodlN?= =?us-ascii?Q?7xkfqz8iYJI9uRDfOTXQ7Q7is5c2E47jsD0YKextGK/x5hIB0Z1a1IVRc3v2?= =?us-ascii?Q?Glleey07A2g7lEkA+MVUd41rbDS3+uWZl2KilwnrVHZhdHGY3bn0CzMK/jBS?= =?us-ascii?Q?SSYhnPkUHanLd+XNdnn1BOkXV6WYf0fCZi6U/xkbxyYsyMyO6WpbO0jSsIvG?= =?us-ascii?Q?l/PJ04DsCvS7QxVt0oyG9/jAMBCD+06g2Dy92Jf0BVehHVEb9eERMjajYsja?= =?us-ascii?Q?YhRXH/OpAHxCZW+hcrCXayshpgNzO1xB1zSiSEOXzZCX44wEX7In4CeXGk/K?= =?us-ascii?Q?wufOGKW290LEWmtyZyJSTbRDSDwKQV3/hxAKNuB7TV+oha2eqRh2UkYzb3qC?= =?us-ascii?Q?4gqIp0k1QKOJF1AGPy2zczwqX0ymy6zGDpMgAkHYPJrE3SJ7m/jz8+PVHLz2?= =?us-ascii?Q?K8M/DbL2IWWk4D9Vzq99JDvTA0OBK+u+5exJRvxP79ICHNumRNZAY5uVW+F3?= =?us-ascii?Q?fZ+glVcE4TFivcIIVE4gKYozpL2U7ptqZKBin97sKY+3iG/c2vN+kXx6LtE4?= =?us-ascii?Q?PhoBm00V9bQCykhG9qKa51LB3A4Vg6MiKl714PiqVVdPjIUOIXAwiHx7DVpS?= =?us-ascii?Q?C+KS1RVvg10Fj9NHyx+vyVWUJeuxKDqiCUcn2DwJu7n6WCo/K9KaIBb7WAIL?= =?us-ascii?Q?EHW3CkdN2/s6cR0v1tGZ02KOY+lcTcXEN/5PUX49TQ7ay9RJ9BLSdvaOYc57?= =?us-ascii?Q?6KlgE0mhMKQnvJfTP+7huT+sE+kIFnTAFBYwI7UMXAYBo7wJMGuEM6EWo0+1?= =?us-ascii?Q?plBl9Tvfk1vYFHsYKwHl/q6hy7MEj0KcHdDsKjBwwxcUw88mfQSxSJXQgbfU?= =?us-ascii?Q?ZUSP3um3wpbFMRbMrJxrh8iCdCUebKS0qC9b2E/jzqsaL4aLGIBtmFbU0Nvm?= =?us-ascii?Q?Mcp1f2vQyV4OJKK8q4a0iXmDxz+490vN3aNhp0zCxBfNSFrVG0j0Lw1hYytw?= =?us-ascii?Q?5njl0kuXBa8tPoCh4UqgN/idZZVeZ5MC4f8ciQESoTdD18K6x2ysASG/Q5vJ?= =?us-ascii?Q?fHtWkCXQpp1Yk5N8OtdOU0aiab08TRTmbC7V+ygWvsln/NWcDNPrIsbTtJH0?= =?us-ascii?Q?CVSnha5YEPBk+RfjkZ0F2/kENmuMoSH8e3GuKhONPdGovuiDLTON/ZnDWHcr?= =?us-ascii?Q?gK/yBHGaAI5sV6GMpmo7LHBtCq3pRjCQubt2x1AFpW7ZWCVYhbHR3N1SuxWP?= =?us-ascii?Q?u7L6SRNoBxxASmTngDNIGkkc8LWszCQ25lHBbdymyyDRrXhbzfFuYe04ZB8/?= =?us-ascii?Q?JU17M5XsjcIMe0xCY28Gc3ydNhbzodZtRDgRZCynKJE+o3mrOz/8NzH3QUrS?= =?us-ascii?Q?QXjmynDOnV6DxbanClI9fueFxiOiOLx2SjzKd2MyMycXipUmWgxfE6fZSU+V?= =?us-ascii?Q?c9dzm35ID7rQZk675u292l7nFH7p6S2bFC6VtQpJV2MeIoG8CxGcljimAysE?= =?us-ascii?Q?3OXSq1P1mpy8Y54kPQG5b7ScRLvKR4izWJNqbkv+?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7ad9b4d-fb50-4636-7186-08db505a4598 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 May 2023 06:54:44.7663 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: WMLW6VnI6mwyNqKvcppP1D/slj/Xa+UjL2iIm3XWDio46x8d4n2ajfhLJyultv1w2Rp1pbCXkGRbh0TDmynUAw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB6832 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Nipun Gupta > Sent: Monday, May 8, 2023 7:18 PM > To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com > Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com; nikhil.agarwal@amd.com; > Nipun Gupta > Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus >=20 > CDX bus supports multiple type of devices, which can be > exposed to user-space via vfio-cdx. >=20 > vfio-cdx provides the MMIO IO_MEMORY regions as well as the > DMA interface for the device (IOMMU). >=20 > This support aims to enable the DPDK to support the cdx > devices in user-space using VFIO interface. >=20 > Signed-off-by: Nipun Gupta > --- > MAINTAINERS | 5 + > doc/guides/rel_notes/release_23_07.rst | 6 + > drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++ > drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++ > drivers/bus/cdx/cdx_logs.h | 37 ++ > drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++ > drivers/bus/cdx/meson.build | 13 + > drivers/bus/cdx/private.h | 49 +++ > drivers/bus/cdx/version.map | 11 + > drivers/bus/meson.build | 1 + > 10 files changed, 1280 insertions(+) > create mode 100644 drivers/bus/cdx/bus_cdx_driver.h > create mode 100644 drivers/bus/cdx/cdx.c > create mode 100644 drivers/bus/cdx/cdx_logs.h > create mode 100644 drivers/bus/cdx/cdx_vfio.c > create mode 100644 drivers/bus/cdx/meson.build > create mode 100644 drivers/bus/cdx/private.h > create mode 100644 drivers/bus/cdx/version.map >=20 ... > --- /dev/null > +++ b/drivers/bus/cdx/cdx.c > @@ -0,0 +1,520 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc. > + */ > + > +/* > + * Architecture Overview > + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + * CDX is a Hardware Architecture designed for AMD FPGA devices. It > + * consists of sophisticated mechanism for interaction between FPGA, > + * Firmware and the APUs (Application CPUs). > + * > + * Firmware resides on RPU (Realtime CPUs) which interacts with > + * the FPGA program manager and the APUs. The RPU provides memory-mapped > + * interface (RPU if) which is used to communicate with APUs. > + * > + * The diagram below shows an overview of the CDX architecture: > + * > + * +--------------------------------------+ > + * | DPDK | > + * | DPDK CDX drivers | > + * | | | > + * | DPDK CDX bus | > + * | | | > + * +-----------------------------|--------+ > + * | > + * +-----------------------------|--------+ > + * | Application CPUs (APU) | | > + * | | | > + * | VFIO CDX driver | > + * | Linux OS | | > + * | Linux CDX bus | > + * | | | > + * +-----------------------------|--------+ > + * | > + * | > + * +------------------------| RPU if |----+ > + * | | | > + * | V | > + * | Realtime CPUs (RPU) | > + * | | > + * +--------------------------------------+ > + * | > + * +---------------------|----------------+ > + * | FPGA | | > + * | +-----------------------+ | > + * | | | | | > + * | +-------+ +-------+ +-------+ | > + * | | dev 1 | | dev 2 | | dev 3 | | > + * | +-------+ +-------+ +-------+ | > + * +--------------------------------------+ > + * > + * The RPU firmware extracts the device information from the loaded FPGA > + * image and implements a mechanism that allows the APU drivers to > + * enumerate such devices (device personality and resource details) via > + * a dedicated communication channel. What is APU? CPU resources that application uses? Then why cpu resources th= at DPDK Uses are not part of it? I also notice that there are more and more vfio-XXX devices in DPDK now: pc= i/platform/cdx. Some vfio related code are very similar. In the future we may need to think= about make them a common lib or something. Thanks, Chenbo > + * > + * VFIO CDX driver provides the CDX device resources like MMIO and > interrupts > + * to map to user-space. DPDK CDX bus uses sysfs interface and the vfio- > cdx > + * driver to discover and initialize the CDX devices for user-space > + * applications. > + */ > +