From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5813BA0547; Tue, 31 May 2022 02:24:35 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 069F4400EF; Tue, 31 May 2022 02:24:35 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id B99E3400D6; Tue, 31 May 2022 02:24:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653956673; x=1685492673; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=FMmzuu/PhMLLxfdKghraNlL9n9/MAdS39IrYiP5iDBc=; b=Tg1xOqjN+cyMpP1p8y1x9ac1M/fxAuN3s6ILFsuelpsufJUzbUBKnzLF i2yBPM3sq0xLhIpY9CfesldBCMaLLfQ0oTI/BOQdxsUmBgnqnj1Uayhh9 +xm1Bkg3QRKrPyrhXG5qn7rK0R1hOwpkU7dywzhGU38tSYF1u+yUOnc3v lmaOZWOlmN3+HAXLiwxED0yX6t72s6ezXOiLc5RsGpTlJ2nwlJfmGKOcP ZvY/otKFuDlbJWFOAQ0C0+vXCyaRb4wCimrOA4sYpzyP0z1E0kRvVAqqi srIF4G4A9id8KmiOW/vTe8EPcW+qRep22094Rifh75oHr1sZgtta/3XYC g==; X-IronPort-AV: E=McAfee;i="6400,9594,10363"; a="257165055" X-IronPort-AV: E=Sophos;i="5.91,263,1647327600"; d="scan'208";a="257165055" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 May 2022 17:24:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,263,1647327600"; d="scan'208";a="580850910" Received: from orsmsx606.amr.corp.intel.com ([10.22.229.19]) by fmsmga007.fm.intel.com with ESMTP; 30 May 2022 17:24:31 -0700 Received: from orsmsx604.amr.corp.intel.com (10.22.229.17) by ORSMSX606.amr.corp.intel.com (10.22.229.19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Mon, 30 May 2022 17:24:30 -0700 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Mon, 30 May 2022 17:24:30 -0700 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (104.47.70.107) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Mon, 30 May 2022 17:24:30 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fgaOoiO+ulDhn3CElCeif/PuhZ50eQuThplawTrWAFnQ8GfDDec171lvhOoYMDc9eoNKu9SwygUtEUtCB0AGNiJVc/PUX0lTmc0p34KbtWYSLNsqz9LVVO0y/z5wOpIQMPqwIVkxXhqThyE/le67EMY3u9qfu/qyG6noNbPBVtq3A8664XIPQku2JoPJtYTaus8XHTeE63Cx5norX9AgZEYYMCX293KLLJKiYrJqI4RYssBSe3LYb8QyTNVDMeNhw81R1S63fZwoqp83/kw4fKlvu9difrLRDFoVCJa37hi58a9qcq6xFjULet7//Rc9a6mbMXaq6CAOoSBt9cr4vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XlGJn7VRlbXB/IMnLdtO/8cQO8FrVHyDCC4yF94ejn0=; b=UjV97FQRwzr5Ut8lA+HL4SlnlEqYsW8Bs1lmSovkcYIknqCD57OScJFv3LoMvMA1oK6SoGeLrTSC8qmVCBrz3yHxcy4//9zVSJpvQN+MV5DFhvs1NHh8ABTqtHC5fGYYmGvjkwjkUUpYGSc3DbJVOV+hSjxWjqjKKpYAzONXkEar9QEDYtZ4siSQTdsQ+SU72sWEK7MHk+BZdrz5bidrfLaeQs6s1mfd7VPdszWgD8e2p7RgJaHoJ0XznJCfqQ2Je0W93gTUTctZvyW5YOjrHTzCR0/JhwjAQFJhrmD0C/9eo0w3hIsWaYA+WjNPXDatVGZLxB4h9ue7dKUN/RkmzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from BN9PR11MB5483.namprd11.prod.outlook.com (2603:10b6:408:104::10) by BN6PR11MB1716.namprd11.prod.outlook.com (2603:10b6:404:4a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5293.13; Tue, 31 May 2022 00:24:28 +0000 Received: from BN9PR11MB5483.namprd11.prod.outlook.com ([fe80::4d94:5d8b:51d7:d8b4]) by BN9PR11MB5483.namprd11.prod.outlook.com ([fe80::4d94:5d8b:51d7:d8b4%9]) with mapi id 15.20.5293.019; Tue, 31 May 2022 00:24:28 +0000 From: "Zhang, Tianfei" To: "Huang, Wei" , "dev@dpdk.org" , "thomas@monjalon.net" , "nipun.gupta@nxp.com" , "hemant.agrawal@nxp.com" CC: "stable@dpdk.org" , "Xu, Rosen" , "Zhang, Qi Z" Subject: RE: [PATCH v4 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Topic: [PATCH v4 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Index: AQHYcaONxl4P3EAYvUeVGXQtV04Iwa04Jgxg Date: Tue, 31 May 2022 00:24:28 +0000 Message-ID: References: <1653535974-1379-1-git-send-email-wei.huang@intel.com> <1653640433-5066-1-git-send-email-wei.huang@intel.com> <1653640433-5066-6-git-send-email-wei.huang@intel.com> In-Reply-To: <1653640433-5066-6-git-send-email-wei.huang@intel.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.6.500.17 dlp-reaction: no-action authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 15d635d1-c6b5-4206-57d7-08da429bec7b x-ms-traffictypediagnostic: BN6PR11MB1716:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: JQfZ6rz1hERwGtHV+4lOC1+W58H4ToPwenDkSS7RuA3mVWoUNdJDRp9tSQ+vAlLjNXA9+aH/yt0/r9R8pLKnGldjXgb8ny6lRTLWW7La3KcRj/ihsF9+z+gR13B5Hf5//iSvTza4H+7QQQUCyY7Pnq2r2m1vM/ZfWFwNRovTDK13TVQf/je35XrCll2P8wZP1XXQl7kam69qxxlkJG9XyA7iA35bstKDQTrZ/CJNZAe/Ayu/UfUelFPofcadf1W1Wdncpy4EN1hJ++6eZUwNcn6YiejkPV9P8ppnjcRlD+QYSDpRgqK5CKEnHLHwifFY7anUhIasnWHMwVyyijM/89Up8a+KR0Xj5LypDOkWiH6cVJH/QrPVY7Q5FkIMBBKt1X/nSHGr1CG/uz7WwQJFYrUPQFctJO6fTFh+9qlHo3g8nZY8cJMT16JmTcwWYrw0fzcEPTBBbzRI++UnLQ5wztWO6mzbFzN3yWR2cf1qc08YV3UjMR2fpiJO7JiHalyFqFAUifQNMmm4V7MnLmGotfa2pOLX8fRiuGIP2Wc7CKRhK4maJI8QQHkg7gQRidAZ4tiN9z9dBqI2sEHayPHdq09LdpqEN/RKhzU9WlI3rZqZtx5tSLF0elQ4VnCEQj23fzvgNSz3DYmj8SagzMKnGr+glHELgGq9oVLGOAjsG3CB70mq3jAf6BxfG3s2cDDW7SkBRKvubjzF251EdEWqJA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN9PR11MB5483.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(8676002)(53546011)(26005)(6506007)(55016003)(316002)(38070700005)(2906002)(110136005)(54906003)(33656002)(9686003)(508600001)(82960400001)(7696005)(71200400001)(122000001)(86362001)(186003)(38100700002)(52536014)(83380400001)(8936002)(107886003)(76116006)(64756008)(66446008)(4326008)(66946007)(66476007)(5660300002)(66556008); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?Rq03xpZukqeOp1GSlii51E0HMnieeKLtbYMaE2xg9NkoNADQ0BY6Qnqw0l2y?= =?us-ascii?Q?En3sxzlV6VtjO3z67sGf2YPtIjVfoZclDfvgdyt/7DCCMpHuQUArZpxLMMHl?= =?us-ascii?Q?CvAb2715FCVMOQgvtr9UqhlU04C8lvNgfLQ1Nc+9YUaW2qTNagIgr61txBkg?= =?us-ascii?Q?IE1DOPGv7mzXcsAsgb8yxmC5QxuCjjmE4DuZMbg8673MpxoNsXqPE8lUsjy3?= =?us-ascii?Q?8oKFkv8038sHTy85l1/rw/NzmzzEM0as8p01jJsIkRKzBRcZi42j8AFy78dT?= =?us-ascii?Q?JDRAa2gLc7/swtAcgI+BpTKCl4wAlyEiH1+i69d170YETxcv99QGT4rP/l9m?= =?us-ascii?Q?QTwRDPxVUzAF9cPoTpSdrB6vs3sfNu/PdniUiP3e/Pzg0bb+vC3XnYRQKORC?= =?us-ascii?Q?aSgutTS7Sq6ca3ry9LwjRLpDOTKAZUY0VazNOso9QNvy2UKJQry7k2Hl+a6G?= =?us-ascii?Q?CeqrgfGZUUANkKd2P3SaEWEwKVhncQCXAR7/9uL6KM07ssbGLnlteYsqY1QN?= =?us-ascii?Q?NBzFiEa31+DOMJHHcQV8Gmq8vSQkVA4I1t47yTil+MWEWP5Lrx6JtyDLq4dk?= =?us-ascii?Q?Cou6eABvmMhwzZ/QWvxiK5y0UQQ5NIaLIuAHliDDn02DM7s5p+xDYUZ2qbGw?= =?us-ascii?Q?G9sz3JuaIDitsggYUD4jLDbSVygc9sYCGa3PpxT+xl5ns/Y4WTlqUEmXpDKH?= =?us-ascii?Q?1I9XXlNNdWbfHL44opfjR+Y/ZGbvM2vKhPKV53jXs3I2yPcQ6a1toG884NpW?= =?us-ascii?Q?kkFjK57dBCPkQiRRe8R05cz8n3tYwqoR/aPu+uaftpZafDvda/Xwa3JXUQ6p?= =?us-ascii?Q?l3L49iM3NgDBrDe/DOBEsp9WthnoYEc3nMkD4ZaEVRGT00cHXtENWRqD2YXx?= =?us-ascii?Q?DLZpr+Wigv7xURhc0f0ivnqi61YGzUnmBki+DWGpIoXJoF4ktcw7WraLS3LT?= =?us-ascii?Q?6S3HWrmyNPISoZQpHPUxqiOpNQWlku+JaefmzFwmuroCuqL4PSHmDibM7M0p?= =?us-ascii?Q?7+k9RTUlvbI1AozuYp4Le79cvNoiP2N56J1aGkOfH7nOxtAGQWggb3Eph+9x?= =?us-ascii?Q?cRk4g8c5QaetTC8y7odojRn7GHAixJMlPr5nOslliixE7bSBhiiKuQsO0qNn?= =?us-ascii?Q?w/40hJkN+wOrXulpDWYw+3W83ZxS+a3GsRNQFkYKggIdtnktWhyjMonskxLM?= =?us-ascii?Q?QhvZi5ewelRIPxrn2RE4aNAmu660oAaSWgPpD+fmxhQpiTDIP4uT2M6BqKM3?= =?us-ascii?Q?qqEB3oK72AYdfEkx71aTsj1gdE5UWfZb7PW1lE0CGj3OqB3Lojx4eybgJ9le?= =?us-ascii?Q?fxItFD4PQIUh4CfEomdR+fnWVi41xTr8ufaG2SMhts5Qz0vu+VCVNG9vyxZD?= =?us-ascii?Q?G4DWQ15UVadOQBpttc6/enWPZkLQz8VPfBMPq/W/JSYg0izs+XqumDP7PTDj?= =?us-ascii?Q?cX39hX0uZ4vrr2ZuqKoVMsIpjgHpLRKN8OdM7YAZgGyB4LcAUjDsJGH8GIt/?= =?us-ascii?Q?LecR2fMJ8R3mi/wG2cJDvumsrSPPtsbuZNwjCpl5N763lbQ/7/HywAMYHmUB?= =?us-ascii?Q?vFy5GVp9GkR3V4kf5I3LLqoS9Zb9t+33qUu5V4r0kE7RUB3niYhkS/pbBUfE?= =?us-ascii?Q?Uy4WgeFg8RsTaJr4/Y8ZmFvrdC/nGyhPAsTZ5elGolQi7REqS8LQtIDmuJ9L?= =?us-ascii?Q?wu1oIzvcmhbo8bZ23ZLXKotkLFgVxXGwCgDeTysWy/4VHdhLwSoDBchWQXjf?= =?us-ascii?Q?Vrx1dk6/2g=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5483.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 15d635d1-c6b5-4206-57d7-08da429bec7b X-MS-Exchange-CrossTenant-originalarrivaltime: 31 May 2022 00:24:28.0904 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: hJ8eYf5lwnTCauXCEem0Xz+V5GN9qWxRETTRLXLDRvxpifVCNPXl4n8iAwUYr5Az1bpBrVmZs76VgtP3POzwUw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR11MB1716 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Huang, Wei > Sent: Friday, May 27, 2022 4:34 PM > To: dev@dpdk.org; thomas@monjalon.net; nipun.gupta@nxp.com; > hemant.agrawal@nxp.com > Cc: stable@dpdk.org; Xu, Rosen ; Zhang, Tianfei > ; Zhang, Qi Z ; Huang, Wei > > Subject: [PATCH v4 5/5] guides/rawdevs: add description of ofs in ifpga d= oc >=20 > OFS (Open FPGA Stack) specification is introduced briefly. >=20 > Signed-off-by: Wei Huang > Reviewed-by: Rosen Xu > --- > v2: update description per Tianfei's comment > --- > doc/guides/rawdevs/ifpga.rst | 105 > ++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 104 insertions(+), 1 deletion(-) >=20 > diff --git a/doc/guides/rawdevs/ifpga.rst b/doc/guides/rawdevs/ifpga.rst = index > dbd0d6e..3dce57a 100644 > --- a/doc/guides/rawdevs/ifpga.rst > +++ b/doc/guides/rawdevs/ifpga.rst > @@ -1,5 +1,5 @@ > .. SPDX-License-Identifier: BSD-3-Clause > - Copyright(c) 2018 Intel Corporation. > + Copyright(c) 2018-2022 Intel Corporation. >=20 > IFPGA Rawdev Driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > @@ -100,3 +100,106 @@ The following device parameters are supported: >=20 > If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and > identifies AFU Bit Stream file. > + > + > +Open FPGA Stack > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +Open FPGA Stack (OFS) is a collection of RTL and open source software > +providing interfaces to access the instantiated RTL easily in an FPGA. > +OFS leverages the DFL for the implementation of the FPGA RTL design. > + > +OFS designs allow for the arrangement of software interfaces across > +multiple PCIe endpoints. Some of these interfaces may be PFs defined in > +the static region that connect to interfaces in an IP that is loaded via= Partial > Reconfiguration (PR). > +And some of these interfaces may be VFs defined in the PR region that > +can be reconfigured by the end-user. Furthermore, these PFs/VFs may use > +DFLs such that features may be discovered and accessed in user space > +with the aid of a generic kernel driver like vfio-pci. The diagram > +below depicts an example design with one PF and two VFs. In this > +example, it will export the management functions via PF0 and > +acceleration functions via VF0 and VF1, leverage VFIO to export the MMIO > space to an application.:: > + > + +-----------------+ +-------------+ +------------+ > + | FPGA Management | | DPDK App | | User App | > + | App | | | | | > + +--------+--------+ +------+------+ +-----+------+ > + | | | > + +--------+--------+ +------+------+ | > + |IFPGA PMD driver | |AFU MF driver| | > + +--------+--------+ +------+------+ | > + | | | > + +--------+------------------+---------------+------+ > + | VFIO-PCI | > + +--------+------------------+---------------+------+ > + | | | > + +--------+--------+ +------+------+ +----+------+ > + | PF0 | | PF0_VF0 | | PF0_VF1 | > + +-----------------+ +-------------+ +-----------+ > + > +As accelerators are specialized hardware, they are typically limited in > +the number installed in a given system. Many use cases require them to > +be shared across multiple software contexts or threads of software > +execution, either through partitioning of individual dedicated > +resources, or virtualization of shared resources. OFS provides several > +models to share the AFU resources via PR mechanism and hardware-based > virtualization schemes. > + > +1. Legacy model. > + With legacy model FPGA cards like Intel PAC N3000 or N5000, there is > + a notion that the boundary between the AFU and the shell is also the = unit of > + PR for those FPGA platforms. This model is only able to handle a > + single context, because it only has one PR engine, and one PR region = which > + has an associated Port device. > +2. Multiple VFs per PR slot. > + In this model, available AFU resources may allow instantiation of man= y VFs > + which have a dedicated PCIe function with their own dedicated MMIO sp= ace, > or > + partition a region of MMIO space on a single PCIe function. Intel PAC= N6000 > + card has implemented this model. > + In this model, the AFU/PR slot was not connected to port device. For = DFL's > view, > + the Next_AFU pointer in FIU feature header of port device points to N= ULL in > this > + model. On the other hand, each VF can start with an AFU feature heade= r > without > + being connected to a FIU Port feature header. > + > +The VFs are created through the Linux kernel driver before we use them i= n > DPDK. > + > +OFS provides the diversity for accessing the AFU resource to RTL develop= er. > +An IP designer may choose to add more than one PF for interfacing with > +IP on the FPGA and choose different model to access the AFU resource. > + > +There is one reference architecture design using the "Multiple VFs per P= R slot" > +model for OFS as illustrated below. In this reference design, it > +exports the FPGA management functions via PF0. PF1 will bind with DPDK > +virtio driver presenting itself as a network interface to the > +application. PF2 will bind to the vfio-pci driver allowing the user > +space software to discover and interface with the specific workload > +like diagnostic test. It leverages AFU PMD driver to access the AFU reso= urces in > DPDK.:: > + > + +----------------------+ > + | PF/VF mux/demux | > + +--+--+-----+------+-+-+ > + | | | | | > + +------------------------+ | | | | > + PF0 | +---------+ +-+ | | > + +---+---+ | +---+----+ | | > + | DFH | | | DFH | | | > + +-------+ +-----+----+ +--------+ | | > + | FME | | VirtIO | | Test | | | > + +---+---+ +----------+ +--------+ | | > + | PF1 PF2 | | > + | | | > + | +----------+ | > + | | ++ > + | | | > + | | PF0_VF0 | PF0_VF1 > + | +-----------------+-----------+------------+ > + | | +-----+-----------+--------+ | > + | | | | | | | > + | | +------+ | +--+ -+ +--+---+ | | > + | | | Port | | | DFH | | DFH | | | > + +-----------+ +------+ | +-----+ +------+ | | > + | | | DEV | | DEV | | | > + | | +-----+ +------+ | | > + | | PR Slot | | > + | +--------------------------+ | > + | Port Gasket | > + +------------------------------------------+ > -- > 1.8.3.1 Hi Wei, It looks good for me, you can add: Acked-by: Tianfei Zhang