From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 51D7A42ABC; Tue, 9 May 2023 13:09:39 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E6428410FC; Tue, 9 May 2023 13:09:38 +0200 (CEST) Received: from outbound.mail.protection.outlook.com (mail-co1nam11on2077.outbound.protection.outlook.com [40.107.220.77]) by mails.dpdk.org (Postfix) with ESMTP id 5F3B3410FA for ; Tue, 9 May 2023 13:09:36 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GWD9H9aZSaGjvSiF+w2EzlhWW4DmZzg1LkBhS+Xf0Q4APyskeQS0QyH6ATiL5KF1N2eVPX50APDlk7EsUw3FKH0+SWqJCgjpfHnIzeb7Z5kUbp4vWKXUBEXRVvxgv2HOIQDFmWWE0IrV7rdCNvLCZMCqYoOwGHiFD9CTOSFIgK5ZRKur+/d07YijyZwST12SC85G8q7kxMfbhDippXg88oW2NAEdhVQ8osXZNzDoWN4qMmYjUp6kuODKRhX01/D1U7HjZXVsiAWasVtGi56EDUTh8AsFU5MKS5mI6XBkCxySqV1b4w08fxn5eL70jIrobWjHtYvt/n/0i6KNn0xfzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R0HvC3RnTBLXTxNtpX6Pvi7G87DcP0b1zp84j/QiPOA=; b=bXe8UUn3SpzEdTCU6wt+bdjuS7EOtUDkDSH6aPEoqNslmj1YV/YXLejtRw5YpCsQayGkTZTB0AfPg5WOrZrQCzilPBn/svAkFm8m4/4N+Wqwtyos8vIi/j1lNH0m2i9VT49fvwQDqdbt+JkJIr8lwow7FcmUCcDG/AUW5A89L8a9IbdufsVBa2VkYTIo1SO8YxU7wz4mDIDHxVCgWLW7gKgz6gJkwMC5l24V+0FoMmcDgg941STl+pRhh3T8/JY8QU+A3GzTVnEvTssBb/9sCTB9K5NGguQOXtQNZjq8LxkOt6TrTPpDbpZ4kwVUSj0mzh0rcYMYALYUONdi8QF9TQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R0HvC3RnTBLXTxNtpX6Pvi7G87DcP0b1zp84j/QiPOA=; b=i3kdX4ptTfxVb1plTqFMFc+4xOfJJ8jmKWrq5djQYApRlpn2vry/6HvlW9iMb9xKiKxl1uIysPFhprUJNcML5PE0hnCtyyr9IqSQigSjykNdVB/6SvFNu2d/oOFtrSaUKIF8RkzwUUCmdQmV+8jfc8NcxYD1ksUm9eAn74UwWJQ= Received: from CH3PR12MB8308.namprd12.prod.outlook.com (2603:10b6:610:131::8) by DM4PR12MB5722.namprd12.prod.outlook.com (2603:10b6:8:5d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.33; Tue, 9 May 2023 11:09:34 +0000 Received: from CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::4e4f:3a93:b7bd:489c]) by CH3PR12MB8308.namprd12.prod.outlook.com ([fe80::4e4f:3a93:b7bd:489c%5]) with mapi id 15.20.6363.033; Tue, 9 May 2023 11:09:34 +0000 From: "Gupta, Nipun" To: "Xia, Chenbo" , "dev@dpdk.org" , "thomas@monjalon.net" , "david.marchand@redhat.com" CC: "Yigit, Ferruh" , "Anand, Harpreet" , "Agarwal, Nikhil" Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus Thread-Topic: [PATCH v4 1/4] bus/cdx: introduce cdx bus Thread-Index: AQHZgZ7Ytfv0pNrrlE2rdPfL+iQqCK9RgwcAgABGWPA= Date: Tue, 9 May 2023 11:09:34 +0000 Message-ID: References: <20230124140746.594066-1-nipun.gupta@amd.com> <20230508111812.2655-1-nipun.gupta@amd.com> <20230508111812.2655-2-nipun.gupta@amd.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_Enabled=true; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_SetDate=2023-05-09T11:09:32Z; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_Method=Standard; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_Name=General; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_SiteId=3dd8961f-e488-4e60-8e11-a82d994e183d; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_ActionId=79c5d602-be5f-4356-bdb2-dfbb2fe4c924; MSIP_Label_4342314e-0df4-4b58-84bf-38bed6170a0f_ContentBits=1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR12MB8308:EE_|DM4PR12MB5722:EE_ x-ms-office365-filtering-correlation-id: f0ab50a6-9e8d-478d-55f8-08db507ddec4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: U0GUqevoNByUajuMniHDTryxmSxaklCSu2zxk3X2dtCBcCTP5SghVktqW27K9AUacdZ/IUX76ZZb1mvo+is/rBdBL1GjE1yOcGsTtVgnIqijxs8TFhwXm8wkXG9734dd9w6UC4/v7kk6xwp4buTxgRSTgcZg9etedI1nJ/vKA1Klez3DohSpGw5yqwS6PEQXYQep9ELk+QaLouT7uA+zntnSH07vnoz66xYAtesLOPiTbnRinkazxlzVUPg/kgludX0YhJGViCynpe95Zz70+Ah2aFkL6Qlj8eP22Tkdeix+5TkWTbz3JZQh47WK9mnClTvGusxHzFOQ93UZuNka2WDlsZl4nsJGvK+w+6/ncltVgeJW/OdU79I8GoyCQU0xldckNRJ15cHzibw8nwKBoSTaGAvQtrb/GzxI6E+wiIKVxDeyeqRxzrPPLQDhndEkIrAkGibLUXdfPG/S5ba99LLKo1wZeDfjy/cIOQkx1NxmOKnyX8zEDT3SKFOoU0v8sMR8rVa5rxdTQzldBeiYTmR3VOHOhXIxO0uBfj1c0H+ry5nHy+1oXsf97ZTMiEglUFnMYaE7NobZ5QtY9hsY+ftkuQjUjrdb5Bcgb3esPMAbkBy7Ygx5y0yB4k75dPYr x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR12MB8308.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(136003)(396003)(376002)(366004)(39860400002)(346002)(451199021)(38070700005)(4326008)(52536014)(316002)(5660300002)(41300700001)(38100700002)(66446008)(64756008)(86362001)(26005)(54906003)(71200400001)(122000001)(53546011)(66946007)(110136005)(66476007)(9686003)(66556008)(6506007)(2906002)(76116006)(8936002)(8676002)(33656002)(186003)(7696005)(478600001)(83380400001)(55016003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Vcd+WD1+3BLKe6DbAehXiBiPbcpfAuaSTmfjbcxHqZB9EqLZUTFRk95qfJJY?= =?us-ascii?Q?cDb3rPuF7edIhlO9o1qCvuSjazVUxgI455gDDZH9YmKQl4dAoF3l1DBkuaMu?= =?us-ascii?Q?qYowtUrZ1WANnraXPAMESRoO2yJLhPYiA7jldQLuTI9dDIx3pbE0R2Pn2YX6?= =?us-ascii?Q?T/67XxTe6vaGikGSJLrRbZb4Og0mvpHE4LyEnjzqFpCN+th4E2lTLYREWBx5?= =?us-ascii?Q?7M9KBv72efUGtxYLTYIdQn/ddhOc+1ODm482ybVrJdHl39sH+6O4Z0eC3ISK?= =?us-ascii?Q?KorgPGbDNQ/ZnN2H4btxb802zTY9bHwsywkRsa8/mcjHFJiGmxj3N424prw8?= =?us-ascii?Q?/9Pbio5UOkTEiEvJ+kyTefbEu1x2kveM7XRzXDzv4gyvtDUocghPXrbUgTpQ?= =?us-ascii?Q?0uiRoB6XDkQopp/Xxvr+Gm3agdWD1euTAN5y7auH9wAtFHNHZhPp5/0mYSrR?= =?us-ascii?Q?s2x9xOWvLSBZfPLdQ/7TTlMCYhr/suQbiaGwZX3zIGuvi8YeKbQcltnuPtkT?= =?us-ascii?Q?o8cI7DoGFADZLxdDtUifWlY5dqziNwG+gjDQNp9BlaNjBMXeu3IQWvLgILUl?= =?us-ascii?Q?VltbrH9NfmWxQDa1e3uuK4zasfJphVyzFa3pJtXnX+PTdi5RyQZMVc+bPkV9?= =?us-ascii?Q?ClhSGtCEVFaGPZLRI4UQ2eIEdlUJeMdp7ljjOgJbQvAhBTBCr3nraUqz1zBq?= =?us-ascii?Q?UpY/6KSyvLl52yGJ5IOo0xrmZ3pS8zaprdouSMKjbCEG5s2gHUHJNEmK9z4X?= =?us-ascii?Q?4CE58WX4cuf/cpnEcJVPoEWGWzadLLpkBaL1pBtetkXgFKoV942z1nn/+5Jj?= =?us-ascii?Q?FGuALVIRpVBYlo2dbPYc1xEjo4zj/cgtD3VomslgEVnE1471KZiUMrxJlkRM?= =?us-ascii?Q?80RLeq1NWa97ImHDJ6h85fDHCsYYH0dZdEIZjP+rqQKjPyL4Z+2h6siR3RjY?= =?us-ascii?Q?G0gewLiLvP8irKXhxnYvaVXMRHHAs6BMozg39h8yoNMsQjTUn5U1TDZ3Nqq7?= =?us-ascii?Q?RHK+HfPdzlxmZ8cb18XOa78aYRDnmFda4uA8LP4iX1mQ2Enh5w0J1GwR98lh?= =?us-ascii?Q?js7FX/4NGi37v3InZmPc8XqnE7f11cpXcsgLRbi8nupDHZ6eKwyb8zR3eMVy?= =?us-ascii?Q?zK1cqnF4E8IE/KIeVPgY1L+GkbHgQqirtUmCnvKZVq11Pllfj074g2HcqhnV?= =?us-ascii?Q?ymE3dSRSkLVJ5zlysym47hRVpuo7U7RTCobjKTbRGS+/sMhy0AlOQSIgvOEZ?= =?us-ascii?Q?aR7ZP5GkuF5idC+F+42Btih2raPvZ+J0K4a5TgSllNnN8N7PpwCaHK5u0PDG?= =?us-ascii?Q?JrIOrjVTpPbjuIb2fxaLbsDBbX87GanhCcjcR0MIlgMziaQ4ICoT/TGgCTAB?= =?us-ascii?Q?KYXl4KQmr1avEvVTKm0ucCT6behHp2H5WLKKRi6cfLRiS8WrxpHqozF40tdk?= =?us-ascii?Q?HK3sesVgvp4YkQyVZoHGGFpvFOop7lr3G468NopoWe9CcfCp5wNoKmuu9tHd?= =?us-ascii?Q?yK7xTuWYtOoGuFu3pb0tB/uMA2O/H3ri53i6oUm1uxTtWmFirKl7MLvqZrxp?= =?us-ascii?Q?qZyIvRl1u045z+X2eOg=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8308.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f0ab50a6-9e8d-478d-55f8-08db507ddec4 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 May 2023 11:09:34.1084 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: nDKoLdqj7cXlpE0WcPCGLZ93WWbIbwW27XRuADXtHUTkXsX7sCn9b48sB5WFotzq X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5722 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org [AMD Official Use Only - General] > -----Original Message----- > From: Xia, Chenbo > Sent: Tuesday, May 9, 2023 12:25 PM > To: Gupta, Nipun ; dev@dpdk.org; > thomas@monjalon.net; david.marchand@redhat.com > Cc: Yigit, Ferruh ; Anand, Harpreet > ; Agarwal, Nikhil > Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus >=20 > Caution: This message originated from an External Source. Use proper > caution when opening attachments, clicking links, or responding. >=20 >=20 > > -----Original Message----- > > From: Nipun Gupta > > Sent: Monday, May 8, 2023 7:18 PM > > To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com > > Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com; > nikhil.agarwal@amd.com; > > Nipun Gupta > > Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus > > > > CDX bus supports multiple type of devices, which can be > > exposed to user-space via vfio-cdx. > > > > vfio-cdx provides the MMIO IO_MEMORY regions as well as the > > DMA interface for the device (IOMMU). > > > > This support aims to enable the DPDK to support the cdx > > devices in user-space using VFIO interface. > > > > Signed-off-by: Nipun Gupta > > --- > > MAINTAINERS | 5 + > > doc/guides/rel_notes/release_23_07.rst | 6 + > > drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++ > > drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++ > > drivers/bus/cdx/cdx_logs.h | 37 ++ > > drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++ > > drivers/bus/cdx/meson.build | 13 + > > drivers/bus/cdx/private.h | 49 +++ > > drivers/bus/cdx/version.map | 11 + > > drivers/bus/meson.build | 1 + > > 10 files changed, 1280 insertions(+) > > create mode 100644 drivers/bus/cdx/bus_cdx_driver.h > > create mode 100644 drivers/bus/cdx/cdx.c > > create mode 100644 drivers/bus/cdx/cdx_logs.h > > create mode 100644 drivers/bus/cdx/cdx_vfio.c > > create mode 100644 drivers/bus/cdx/meson.build > > create mode 100644 drivers/bus/cdx/private.h > > create mode 100644 drivers/bus/cdx/version.map > > >=20 > ... >=20 > > --- /dev/null > > +++ b/drivers/bus/cdx/cdx.c > > @@ -0,0 +1,520 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc. > > + */ > > + > > +/* > > + * Architecture Overview > > + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > + * CDX is a Hardware Architecture designed for AMD FPGA devices. It > > + * consists of sophisticated mechanism for interaction between FPGA, > > + * Firmware and the APUs (Application CPUs). > > + * > > + * Firmware resides on RPU (Realtime CPUs) which interacts with > > + * the FPGA program manager and the APUs. The RPU provides memory- > mapped > > + * interface (RPU if) which is used to communicate with APUs. > > + * > > + * The diagram below shows an overview of the CDX architecture: > > + * > > + * +--------------------------------------+ > > + * | DPDK | > > + * | DPDK CDX drivers | > > + * | | | > > + * | DPDK CDX bus | > > + * | | | > > + * +-----------------------------|--------+ > > + * | > > + * +-----------------------------|--------+ > > + * | Application CPUs (APU) | | > > + * | | | > > + * | VFIO CDX driver | > > + * | Linux OS | | > > + * | Linux CDX bus | > > + * | | | > > + * +-----------------------------|--------+ > > + * | > > + * | > > + * +------------------------| RPU if |----+ > > + * | | | > > + * | V | > > + * | Realtime CPUs (RPU) | > > + * | | > > + * +--------------------------------------+ > > + * | > > + * +---------------------|----------------+ > > + * | FPGA | | > > + * | +-----------------------+ | > > + * | | | | | > > + * | +-------+ +-------+ +-------+ | > > + * | | dev 1 | | dev 2 | | dev 3 | | > > + * | +-------+ +-------+ +-------+ | > > + * +--------------------------------------+ > > + * > > + * The RPU firmware extracts the device information from the loaded > FPGA > > + * image and implements a mechanism that allows the APU drivers to > > + * enumerate such devices (device personality and resource details) vi= a > > + * a dedicated communication channel. >=20 > What is APU? CPU resources that application uses? Then why cpu resources > that DPDK > Uses are not part of it? Hi Chenbo, APU's are application processor unit and are on-chip CPU's. So in short APU= s are CPUs only on which applications like DPDK are running. Thanks, Nipun