From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 81ABA42ABC; Tue, 9 May 2023 13:25:50 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 00CAD410FC; Tue, 9 May 2023 13:25:50 +0200 (CEST) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2067.outbound.protection.outlook.com [40.107.94.67]) by mails.dpdk.org (Postfix) with ESMTP id D6EA4410FA for ; Tue, 9 May 2023 13:25:47 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CTScdVP7bWsqbMl2pgNZIIrQgI1hYjm12y5Vpp/VcbbGrGm/VGnWpqkLlDnXxxOtk78kcd29S7UK5Ig9EUSfEA9yz4fJHOBeS8QBwPwWn/qgB0KpiXvHjgAB5IGGDbJbwzSiZ74haFTLBsv30okzviVT+V6gzSZ6qw1SXQRTgFRoBvkJNLwD6p4VFSPnFdwvNJ2/rajFLyU4YQP9rqDKOcPasoknU/F9USjO0R8krkG5b2ZEB3FCv2+9COw7Mow+mLyjk/sgQ5TkYZI7L3VSZq9ZZ5sYn7mSERh7Jr25CtEHlUBoNaQ3HC9JAILB2L0JQa9/iS4IZ0WuNd3xyV7faw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VYf+dfoQ8MCijTzvr06zKX+T0YYi/rdQTNrMaY8+ShA=; b=fJ8lVsl62QEoBExdnYOEqU8yPFLFQHcE4R9HuvRSP3hKKRQ5Ahny+pQwxZOSW+/z7+eqO4yCRMQa9OZq3d2oYeo58ZtEJ29V8556dv0MH01cL/b1y1zablC64Mi2O2/nshXlQMFQSOyZMpJwbUuJu/acxaFVDdBMZQVRHb4OD9/qhFzBZ2noWgKse0Dk2MyePmbG1sVgx2Og/CeI+UTAeq1db2oTJKKNPwIz/a1bf7eGS47yRJ79lZ+/2SkjhWXPXFo1LGXAgRg3RxEleJcGJL3YYdWDsNYn6E0Icyz9HIWFA3MaShUxBlxt3uG9yKH3/i8QhV1fZe1kS+UJTZVYJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VYf+dfoQ8MCijTzvr06zKX+T0YYi/rdQTNrMaY8+ShA=; b=f2uQlw/hoz5fh8t3ak9O/A2q/KiNrQ3ngrTQyr8XtA81AdfK/rK9K57EKYjHcpcnIw+PJiktWTZxVT4/WTrhpjrDZXQDaasJAISD//zjOrGhMsOJk9LOvJFmfqnwWexZiZtFjpslrNaxcXesPvOXLhOH609j7tKAQTk6pN2Mz0w= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by CYYPR12MB8749.namprd12.prod.outlook.com (2603:10b6:930:c6::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.33; Tue, 9 May 2023 11:25:44 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::e818:77ea:75b5:f8cc]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::e818:77ea:75b5:f8cc%5]) with mapi id 15.20.6363.033; Tue, 9 May 2023 11:25:44 +0000 Message-ID: <9a7b7495-3d27-dd0e-03f8-43c0795bbb62@amd.com> Date: Tue, 9 May 2023 12:25:38 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: [PATCH v4 1/4] bus/cdx: introduce cdx bus Content-Language: en-US To: "Gupta, Nipun" , "Xia, Chenbo" , "dev@dpdk.org" , "thomas@monjalon.net" , "david.marchand@redhat.com" Cc: "Anand, Harpreet" , "Agarwal, Nikhil" References: <20230124140746.594066-1-nipun.gupta@amd.com> <20230508111812.2655-1-nipun.gupta@amd.com> <20230508111812.2655-2-nipun.gupta@amd.com> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0053.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:152::22) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|CYYPR12MB8749:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f1760e8-81d6-40f5-9c8d-08db508020de X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: p2o06VcWbTdBsNLPbfkVbih1AV92wT1aG+UNSWQytkjR0BBE3olTMYeclm3C0VZ/cYdA3YzW+/GugwNhe5HwwkpBHRASpRdGrTK1xfGj+PkM92GgyDOXFR1Qmn6m1sr3PYRXM/bJtApwQWTEOmOEREmexl1MBXs2pfEh8UmIIDcsEnTjC/CcLKMbbe/hYvLg5zyXiQarigzwgx5FUyQ5h0WI1hrZYXZ/lOXUJQiMll420fY6wJpT0+YlR+0OT+AeOomEshwqQuOG24eyw5DlTYlLKsY88DVvXVCj9d/fK9RVlSRUAnSiNxjjBRdzbcTaKfkV0KaxbbWD65j6xTgvj/4+USPZ0ovGcnB6kdYncw15eBGYilRVmcG8hLXuhRj5hP+XD4IEyJ2DinZ2YEeO34ns9//NOj1NvArvR8svVmv2ctNhwbBDHV6aJ3+Npu0oTtqHxiXnf9cehFPEZIxvEOevv1JnO9wscKsMFlAml1/xyizacRSJRbe1WhPUXEuLqnS1eeSmzBegPXA9GpbIKXOw1D5DVSPiNCcZc45S2JonWpWQa+6ayTb25lRvamLknfUsmBLMxdaYBNDoBt5aaq8VIho2ApmuFTk2MF9ADbv3sgacHbN0rkTbwm4Wz0GnqsagR2tyYMY/zM0I8TARVQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(136003)(396003)(376002)(366004)(39860400002)(346002)(451199021)(2616005)(2906002)(186003)(36756003)(478600001)(44832011)(6486002)(6666004)(6512007)(6506007)(41300700001)(26005)(53546011)(316002)(83380400001)(38100700002)(54906003)(110136005)(31686004)(4326008)(66476007)(66556008)(66946007)(86362001)(31696002)(5660300002)(8676002)(8936002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dEFaQytNT296anJHcCtqaVdxbjJaZ2thNjBsVnFiemNCT0xPc2huQ3R3d3No?= =?utf-8?B?UHE0VGRnVm9DY1ovTDBKU0xrcmxRMGpkVXN0d3FJNjVYb3NYeG95VTNleVdF?= =?utf-8?B?aE9YVFluazdRSE9jTXpWVytWRnU4SGRTeWcweWw1b2E0NC8yTEhuWlVaNVJs?= =?utf-8?B?ck1JVTlzeWxkczFqSzU3NzlKQWcxellqMXVnUEhkcFJkNDJUMWNIaUdWUGVu?= =?utf-8?B?N29rMFRIbkt0NEttSXkwNkkrbU0xS1d6OWVNendoejVKWEVNd0p5QVUvM09E?= =?utf-8?B?MWYwL1FoTzlSM1IxV25uV0VGVkdKN3hBQ2tta0Q3NUozSlVWZjR0Zkc1L2hi?= =?utf-8?B?YWx2NUtLVGg2azZRMWNzUnN1SzFwMzgvR29PcnBCYldwY3B4QmM4VWRmMWRM?= =?utf-8?B?akpleElOU2tLSkw1blAwajZPRE43emhBaDFOV3JMOGdRdDJHRFJUS25SRkU1?= =?utf-8?B?RkJydXlzUytsQzN0SmYxT0lWQjVvdUc4Z2tMcVNKUCt1WStCd2JLZHJxK0Z0?= =?utf-8?B?TzQyRzlCckNLSGxSZU8rOE93Q0s1enVjNkxRbno0VWVEb2FxYzJEUVdEL1Ez?= =?utf-8?B?YlNyNGRjQTlpcDRxLzFEN3RpSkJtTXdneUNLWVkzbjEzWG1vR0dPM2dRdlpV?= =?utf-8?B?d3J1Zjd6ZVNaVHJ1Q2xIaER1ZGswTDBIV3hPdjdlV3c2MlphYUJYZHgzVE55?= =?utf-8?B?SzUrenFaUmV0R1E3cExVZ3JwMVJmaTcyTmR2cGN6VjVBaWlQOURxUyt0cUpM?= =?utf-8?B?K3JlWGhPbk5xRkw5cGJoOTI3MWdhYTBkcHJTRlAzREhXdmI3WUowcHhBUmZ1?= =?utf-8?B?TDFDazhBR2lJQ3hzaUFpY3RDdjkwbWVEZG9jWXJYVnZvajhXNjAzR3hqL2h0?= =?utf-8?B?TEliNFBNMGt1MEFNRnRHeDIzQUd1TXUvdnhBUXpUL3dkNnI2M0g1ekFzQ0hp?= =?utf-8?B?bWM4Uklpckx3RE1PalkzVkxUa0dWQW1mYWVlb2JFVkNXR3JYYzZFWElGazRE?= =?utf-8?B?bUwzUXljcFRsY1lVZ0k4VkNYRVg5c1FjVHBoL1NjeGRtNlNZUkxDRVd1OFRN?= =?utf-8?B?Tkc4U1ZwdmZMNEYySVNRTjJubU9qSjI3a2hveUpBNHZEc1NSa3RhcFZKNWcz?= =?utf-8?B?YVplT3NOblViZ3VMV01jYnBvNWlHWjE0dVo4YVFrR1h6bHF6eHpWOStnM2kz?= =?utf-8?B?NTdtUDVCa2JTUGlhYWE2TmFCa1hjT2RBSTZSOEZESU5WeXBOMXk3YjVZYjZv?= =?utf-8?B?MjNleFBsNERITlI3bTIwT1ZueWNPZkZiV0lxSVUrYW9kNmIzK25aSnBBd0U4?= =?utf-8?B?cnZMeW9iNmx1UlRLYWkxK1YzSW1TRDRsS0VUVDlWdHBhdlY0MzhBMDYzU0V6?= =?utf-8?B?aFo3VFBlaHErWno1M09HNVJyREc3bmVNMCtsSUpBVmdJWkxwYVcrcFZFMHRk?= =?utf-8?B?cEc4Um1KU3Vuem02SmxQeDc2dW9PRXJMUlA4aTdNS1JScmFPRS9yNFhZcUJy?= =?utf-8?B?Ny93WmgxZUhQcThzMHdzUU9qTWh1eWlvaXp1VUtZK2N1bVpPUFVieGEyNnIy?= =?utf-8?B?czlzN3pSTFcvQ25halFzd1dleVU1MXlmZjgyM21IeStLeFJRZkk1NlBuMWpM?= =?utf-8?B?Z0NnQjMwK2lxZkJKM3BuSzZwNE0vVzk2WDcrTGlWdnNaaWdzSytGVGhZdzhw?= =?utf-8?B?Y29MWHBBYkI1TW9jWGhwc3QrSVp1UEVjc0tESEhzdTRFUVBhd2djMnRSUmk5?= =?utf-8?B?d0orWCtyMlJRTkJIY2MvSGI3VEJBMkgyZUJmalUra1piU0J5U3cvT3VxUWFi?= =?utf-8?B?T1hXNGdZWnhFc09PaWpKMlBHQlJkbUlQSWNpUjRQdFlOZUdmaWpCaHlvUkZL?= =?utf-8?B?R2xzU25oQlpoNjRZa2pCQmNMWEplU09GNllvQ0I3bFFOQTU3N09ERlNpMkto?= =?utf-8?B?Tys4c01jL2Z5eUxHaHZYdzBnYklXZEw1LzRTT2Znby9VL05leGFTclhLZ3Ja?= =?utf-8?B?WFV5NXhhdUszbGl5VUZwdy9kTGFjUHMyTkRtRUViclhxOVJoLyt4TGgwWklu?= =?utf-8?B?TEFhUlh3RjNZZU5KcjNhNm44SVdJWm5wcDJwWVE3Y1ZneW5HNGVMeTdIVXlK?= =?utf-8?Q?e5Y+nHCV5AIfVBcrexDR212Aw?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f1760e8-81d6-40f5-9c8d-08db508020de X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2023 11:25:44.3462 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +3PHrS2/S6ol8rMalyyaucUVY+29cgRpYs/3Sf3gZdaWD8KKfWQCsqaxcNTSage+ X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8749 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 5/9/2023 12:09 PM, Gupta, Nipun wrote: > [AMD Official Use Only - General] > > > >> -----Original Message----- >> From: Xia, Chenbo >> Sent: Tuesday, May 9, 2023 12:25 PM >> To: Gupta, Nipun ; dev@dpdk.org; >> thomas@monjalon.net; david.marchand@redhat.com >> Cc: Yigit, Ferruh ; Anand, Harpreet >> ; Agarwal, Nikhil >> Subject: RE: [PATCH v4 1/4] bus/cdx: introduce cdx bus >> >> Caution: This message originated from an External Source. Use proper >> caution when opening attachments, clicking links, or responding. >> >> >>> -----Original Message----- >>> From: Nipun Gupta >>> Sent: Monday, May 8, 2023 7:18 PM >>> To: dev@dpdk.org; thomas@monjalon.net; david.marchand@redhat.com >>> Cc: ferruh.yigit@amd.com; harpreet.anand@amd.com; >> nikhil.agarwal@amd.com; >>> Nipun Gupta >>> Subject: [PATCH v4 1/4] bus/cdx: introduce cdx bus >>> >>> CDX bus supports multiple type of devices, which can be >>> exposed to user-space via vfio-cdx. >>> >>> vfio-cdx provides the MMIO IO_MEMORY regions as well as the >>> DMA interface for the device (IOMMU). >>> >>> This support aims to enable the DPDK to support the cdx >>> devices in user-space using VFIO interface. >>> >>> Signed-off-by: Nipun Gupta >>> --- >>> MAINTAINERS | 5 + >>> doc/guides/rel_notes/release_23_07.rst | 6 + >>> drivers/bus/cdx/bus_cdx_driver.h | 201 ++++++++++ >>> drivers/bus/cdx/cdx.c | 520 +++++++++++++++++++++++++ >>> drivers/bus/cdx/cdx_logs.h | 37 ++ >>> drivers/bus/cdx/cdx_vfio.c | 437 +++++++++++++++++++++ >>> drivers/bus/cdx/meson.build | 13 + >>> drivers/bus/cdx/private.h | 49 +++ >>> drivers/bus/cdx/version.map | 11 + >>> drivers/bus/meson.build | 1 + >>> 10 files changed, 1280 insertions(+) >>> create mode 100644 drivers/bus/cdx/bus_cdx_driver.h >>> create mode 100644 drivers/bus/cdx/cdx.c >>> create mode 100644 drivers/bus/cdx/cdx_logs.h >>> create mode 100644 drivers/bus/cdx/cdx_vfio.c >>> create mode 100644 drivers/bus/cdx/meson.build >>> create mode 100644 drivers/bus/cdx/private.h >>> create mode 100644 drivers/bus/cdx/version.map >>> >> >> ... >> >>> --- /dev/null >>> +++ b/drivers/bus/cdx/cdx.c >>> @@ -0,0 +1,520 @@ >>> +/* SPDX-License-Identifier: BSD-3-Clause >>> + * Copyright (C) 2022-2023, Advanced Micro Devices, Inc. >>> + */ >>> + >>> +/* >>> + * Architecture Overview >>> + * ===================== >>> + * CDX is a Hardware Architecture designed for AMD FPGA devices. It >>> + * consists of sophisticated mechanism for interaction between FPGA, >>> + * Firmware and the APUs (Application CPUs). >>> + * >>> + * Firmware resides on RPU (Realtime CPUs) which interacts with >>> + * the FPGA program manager and the APUs. The RPU provides memory- >> mapped >>> + * interface (RPU if) which is used to communicate with APUs. >>> + * >>> + * The diagram below shows an overview of the CDX architecture: >>> + * >>> + * +--------------------------------------+ >>> + * | DPDK | >>> + * | DPDK CDX drivers | >>> + * | | | >>> + * | DPDK CDX bus | >>> + * | | | >>> + * +-----------------------------|--------+ >>> + * | >>> + * +-----------------------------|--------+ >>> + * | Application CPUs (APU) | | >>> + * | | | >>> + * | VFIO CDX driver | >>> + * | Linux OS | | >>> + * | Linux CDX bus | >>> + * | | | >>> + * +-----------------------------|--------+ >>> + * | >>> + * | >>> + * +------------------------| RPU if |----+ >>> + * | | | >>> + * | V | >>> + * | Realtime CPUs (RPU) | >>> + * | | >>> + * +--------------------------------------+ >>> + * | >>> + * +---------------------|----------------+ >>> + * | FPGA | | >>> + * | +-----------------------+ | >>> + * | | | | | >>> + * | +-------+ +-------+ +-------+ | >>> + * | | dev 1 | | dev 2 | | dev 3 | | >>> + * | +-------+ +-------+ +-------+ | >>> + * +--------------------------------------+ >>> + * >>> + * The RPU firmware extracts the device information from the loaded >> FPGA >>> + * image and implements a mechanism that allows the APU drivers to >>> + * enumerate such devices (device personality and resource details) via >>> + * a dedicated communication channel. >> >> What is APU? CPU resources that application uses? Then why cpu resources >> that DPDK >> Uses are not part of it? > > Hi Chenbo, > > APU's are application processor unit and are on-chip CPU's. So in short APUs > are CPUs only on which applications like DPDK are running. > There are two sets of processing unit in the SoC. FW runs on RPU, everything else runs on APU (Arm cores), including OS, DPDK etc..