From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 13621A055D for ; Fri, 27 May 2022 05:19:45 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E9BFB427F0; Fri, 27 May 2022 05:19:44 +0200 (CEST) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id 5364040DF7; Fri, 27 May 2022 05:19:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653621582; x=1685157582; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=zUTKDxEV9Tl8J5L5SSOqHopJavEZ7YCc1sLnAtq+pUU=; b=Oo7yEXaMBboMMGAO3l0Z++KcVeCuVJJ+SrM1MBD1d80DcLq4e4xUbt4/ +tK+UxkIK95In5Nhlx7lWAD586DXAm2PpnMr/FlRfC3dYlFB9SEXfOAcl DIy5nCfx6doaGnpo2xHgjrHEzcU3JzeClWzcQ9Umdigj3E0Z72+Fv6lcp G1BHYQKwENboct6wZ15/jC8/hBlW4Jnh98voyPu/BtblBcoiEey6KBfp5 UyEcNEyiebLX+uTp74Sc2KWNlKFiur6hN/17jVbSdYMDDRC8+pwWDg/wM 8WnpdFbsCBn0CIY5Lh+w+13ntpWg40HU5W466yIxN54f79fjLcO9kTwKT g==; X-IronPort-AV: E=McAfee;i="6400,9594,10359"; a="335011268" X-IronPort-AV: E=Sophos;i="5.91,254,1647327600"; d="scan'208";a="335011268" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 May 2022 20:19:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,254,1647327600"; d="scan'208";a="527759375" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by orsmga003.jf.intel.com with ESMTP; 26 May 2022 20:19:40 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Thu, 26 May 2022 20:19:40 -0700 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Thu, 26 May 2022 20:19:40 -0700 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (104.47.56.45) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Thu, 26 May 2022 20:19:40 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K0zbOVKZG6LNjULP0I6pnTs1kZKrgjoVctQ5wWNeWq7tY4YH9tMSRtsMGBGOyIjUk9BWp5VpSDqmBD0UOV5+m8jz/dS0vXqkZ0B+Ld1aZILTFxjH/mn9UpdMtL5Cj6tawoL+EvidjZu9sTd2Kpv8PqRhsZn2barE+maeAAjagHNizkhLiavbNX1YuDYi+QAhAguwd2volgD0HRkf0ruxEe7KSv+SV0UGlHPLG8CZ+ZsmPWJJwZS7AAeQlOj2p7uy9tTLIqvehQC30S+05BxxmCIYh9yCN8UMrndpfcddHjs+BfOFkfpHtBGzllS5SWgen789f+5XOea09OaaxejbKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=MvTfHfoUmKGW6oLOX9qlBXlBpA01cTSyVBzRA0bsKcA=; b=Mwl492yAUTgbALDPKQOGm0tvI2Xzb2KGeADzrROlDq6afVjAasqLClbNsqhws8Ed3rdbdmPVNW+XIQZW0a3FO8ti1E6Tj2ePFeSU+P94djwu7WbcbixMjgWAXg2mAr82qhS+8l0AVdLnsNpHbazuvJCy0xI58Rkpq5cdx20iAZ/klr2fNif3DTzFCDuutfPXRGq9pYH+/Eu1bDUFNrvQ0fWgEwSQQTgD5A2+yf/T/g0iQYUN9zAM2IpcTqnMBktqQRmWymLwJiGdkU4uQWvboLL31ru48qbwi/ghsRpW4wM3cdNbEliFEXZ7H2+sheE0OJnJhxgFJ/otV4b9Jjd1rA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from BN9PR11MB5483.namprd11.prod.outlook.com (2603:10b6:408:104::10) by BY5PR11MB4088.namprd11.prod.outlook.com (2603:10b6:a03:185::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.17; Fri, 27 May 2022 03:19:37 +0000 Received: from BN9PR11MB5483.namprd11.prod.outlook.com ([fe80::4d94:5d8b:51d7:d8b4]) by BN9PR11MB5483.namprd11.prod.outlook.com ([fe80::4d94:5d8b:51d7:d8b4%9]) with mapi id 15.20.5293.013; Fri, 27 May 2022 03:19:37 +0000 From: "Zhang, Tianfei" To: "Huang, Wei" , "dev@dpdk.org" , "thomas@monjalon.net" , "nipun.gupta@nxp.com" , "hemant.agrawal@nxp.com" CC: "stable@dpdk.org" , "Xu, Rosen" , "Zhang, Qi Z" Subject: RE: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Topic: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Index: AQHYcLBXTrK1wUreokm55a6iILEJeK0yDqDA Date: Fri, 27 May 2022 03:19:36 +0000 Message-ID: References: <1652862549-13131-1-git-send-email-wei.huang@intel.com> <1653535974-1379-1-git-send-email-wei.huang@intel.com> <1653535974-1379-6-git-send-email-wei.huang@intel.com> In-Reply-To: <1653535974-1379-6-git-send-email-wei.huang@intel.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.6.500.17 dlp-reaction: no-action authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6a25be74-42fe-4f7b-4c21-08da3f8fbaa5 x-ms-traffictypediagnostic: BY5PR11MB4088:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: cb+bMhpycd1Y3P6egtlejuOsxRdzk4DxEi9pQqAyxowt/NrQ5hheV4fTYIBbES2c7tj28WIS8F8jIBcuuAkcinelxDPN+J+c5yiVtzRhEMVxlO25q/bXSSdgQvB0sogLBHHlgjohvIAQKHjmD732fDxI4rQVW2m8chkv/s/6nkTcQTL1oNzWMICkwS4CKj4QO/k0TCRYH86ononVB6DYH20FLuDTHejYfzxDyf6ffg7reTs2d+0bGuck8t2TnDM9tGOHWnRWcCXSqyYm1uLcl3cTFED61MutHt2tumzVYQlSCHMdPAfWCy5N2ubVypw1fj2s/L8vwFtsdhbYvwcq9nhsQ0Z9qgMIDChUk2O4LDrMyvxuemrUeFoL3yrZ7QnhBZedHx98OFsF4vI8pz0ATwG48/NSE/2CKrNj2lXEtAsRC8L4Yn0cQSP2Pn+KKXXvYBNLHO2BA23zR88+n7DIzCFyUZenlvHSWvlUbqTKC0bRJua5FWfA+ZE6Flvz9eGHGfTLMHVbVbLVwFGP9QaZOYk4mIVCZI6Ul99db901BuUgG9JOetpHChifxVn6jV8YhUsLMGl1bW6Xoh44zuk4s0Gbw0WO2JoUoBOS1/S23ECg+EqJ5VXpiHxD7TpJ925i3x35hRwnDhUjhhmv7cvOxZbAeFMYhm76Wwx1Du3H+P84/4WEfatr2r9BqK2wZFusvVo95v0MlcBV5At9NqOmTg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN9PR11MB5483.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(26005)(5660300002)(52536014)(83380400001)(38100700002)(38070700005)(8936002)(82960400001)(122000001)(110136005)(54906003)(186003)(33656002)(508600001)(107886003)(6506007)(9686003)(53546011)(71200400001)(7696005)(66556008)(76116006)(66476007)(66946007)(4326008)(8676002)(64756008)(66446008)(86362001)(2906002)(55016003)(316002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?AlNn2ISaTe8P9NFsSE63cD30RTbF3dv995wqXTXMuxSnrVBpp7RZ1yGC+F/Q?= =?us-ascii?Q?JUbg8TWh6RUJD7Dg29NjYL1x078q64XC6z2aHcXs8pwejTh+J8VFQ8YAp1Ij?= =?us-ascii?Q?dGstxWGubv940wLBaLYpcQa+hq9mA/qwXaDYAJqxPd7WNTD2s/uHww3FzN00?= =?us-ascii?Q?4rDYAdYnlOj85fs0ZvH7NDAOofYY5aSewUlehrFRr/yD16C8onjtPD0NRv6h?= =?us-ascii?Q?DbLeVMLDJdwjYZhy9NpvfViB62zps1rpi3gc1iAjxU5DRvT8YVXV3Hmo2frm?= =?us-ascii?Q?GsIc5RdBaPWohJAnZzK1tLVaLuGiPpFQQ1SHd2SIBXB2FfoDZVwcyyMUhhs2?= =?us-ascii?Q?KX5fBp8YLZKi3M3utfPW7sSBx6xCmC6/1YtuaTRnjoC5TdTv5bfqDZagYkCI?= =?us-ascii?Q?WA5IwYcqK7Rdb1WuoSYukx9mLLAN9jIx/ZVvGwC1ZJ+fJEUfyi3lvfaO9DQE?= =?us-ascii?Q?LLyMaJUs+Bh2t9GD4W1xwIcd2Ti/nqshSDY6WLqjQ+iuY3exjJ3IwzFlrZMU?= =?us-ascii?Q?ihWcaGAWJu8qFZI8KyjbKZYM2uUQSfUga0h7AQBK+qFKOgIUtxgqyvKkvRyV?= =?us-ascii?Q?gUZE4Gkw8vD5rCREsa92dVqdva7/FplG3QtMOuQCHgrH546BR/rmeNab5y42?= =?us-ascii?Q?xbsBUPUPxYioLiIraT+J6LTFTRVgdbzUXpz++Vxm/E8L0JL3NcQ2Iit850r3?= =?us-ascii?Q?0V/9d2WaD1r2Z9yt8Zd6XYWnJsKQJ5F08EybbXuXjiA5EwMDIRShYpSv5Uuo?= =?us-ascii?Q?Mv5xISp3J59OdeZ3UFajFiQSSWDPa4DDig7UnQGP0tQIIfyozBmvclo5cn5b?= =?us-ascii?Q?GDteEjaB4fhvZ76mrr5RrM1gc2R1K2d5a+55AVLPIFhy+pYoQRKiwCHf1qnD?= =?us-ascii?Q?IOFXVjPiIPFTa7kkGKZaxLGVX4U00Zyh/Sg3lJibaAqGa4MWFMR5yqRQeYXm?= =?us-ascii?Q?rIoQa6J0tVBsopiNqzIS3R93dRLGlnQPkrY3LgMfvdtuOxmxpWgFNOCVnnBJ?= =?us-ascii?Q?nsDXIOeI85wAF2lhlyArPqat3ft2Ewoq6sxigZ8LZ3OqvoX/y3qy5YKapBXi?= =?us-ascii?Q?4ICov5tliNMoGrPUC+V39ECa07/pfbr9haWOjSL3Q5ZBUbmfpi/JC/5ipk3m?= =?us-ascii?Q?FfgBZFNODJRolcowj/8Apdo4iu+2KyKJGE+VRR3Y87hRlhpgMoyTklHr7WyO?= =?us-ascii?Q?arThS4nJLGqXDMIWEQ9zWVMGHWOeXZi+Whhnb9X9rrlhMb7sy8+2a/OX2HW5?= =?us-ascii?Q?0N41YVnL1VhI56z/NRoAirJNE6zhaHfIei4rGTSYSZFlC9epIma72tk1onss?= =?us-ascii?Q?96DQLs8R7zefhsdvegXxe6gvdWk4nqFt59Hyoz1hV7aCgGllqsw1iUH7pmzr?= =?us-ascii?Q?++58Jgv7dYdIkHLG1Qwij4ngWhIg0KRHnFWk/TFLvrCOqZvSj1d7y8PL8ZiL?= =?us-ascii?Q?Eg8+386HNwCchVrlF1Bx1bXUpvjn7cWcilGCuuwHj3x+LyJ1ENzcYDX8ArHJ?= =?us-ascii?Q?gOHQpsuF3Fa15oktFPT7xHnzQPgH9snWeX2sthXmrjv5etuoWWJtLl997w9g?= =?us-ascii?Q?LQVM1MidGPHglsN9cioapfIsgGN/lvd/LwKsALLOaJgOKKO72u/eHGnbpxy4?= =?us-ascii?Q?YFj/Uup6d5O4S+AEkamWi74V7lz5Q1ollC1Q0ev8xLEDgfbZfxnHIms07EPA?= =?us-ascii?Q?z8dhFnX8sT/yGgF5EanjJCYykkDKu9bN8uC2m/rfgvdqu/LtaGgm7IjhAi7S?= =?us-ascii?Q?zaK9aGeyvQ=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5483.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6a25be74-42fe-4f7b-4c21-08da3f8fbaa5 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 May 2022 03:19:36.9990 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: UyFpk5CkoRssSFQ9GnHI3oW8nEtlz55nh9agVQWuK9glG9NmEXLnYbj9TT5Ec3yAZC6YFFPyW2HpPv7hu6MqAg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4088 X-OriginatorOrg: intel.com X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org > -----Original Message----- > From: Huang, Wei > Sent: Thursday, May 26, 2022 11:33 AM > To: dev@dpdk.org; thomas@monjalon.net; nipun.gupta@nxp.com; > hemant.agrawal@nxp.com > Cc: stable@dpdk.org; Xu, Rosen ; Zhang, Tianfei > ; Zhang, Qi Z ; Huang, Wei > > Subject: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga d= oc >=20 > OFS (Open FPGA Stack) specification is introduced briefly. >=20 > Signed-off-by: Wei Huang > --- > doc/guides/rawdevs/ifpga.rst | 114 > ++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 113 insertions(+), 1 deletion(-) >=20 > diff --git a/doc/guides/rawdevs/ifpga.rst b/doc/guides/rawdevs/ifpga.rst = index > dbd0d6e..8809bfc 100644 > --- a/doc/guides/rawdevs/ifpga.rst > +++ b/doc/guides/rawdevs/ifpga.rst > @@ -1,5 +1,5 @@ > .. SPDX-License-Identifier: BSD-3-Clause > - Copyright(c) 2018 Intel Corporation. > + Copyright(c) 2018-2022 Intel Corporation. >=20 > IFPGA Rawdev Driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > @@ -100,3 +100,115 @@ The following device parameters are supported: >=20 > If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and > identifies AFU Bit Stream file. > + > + > +Open FPGA Stack > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +Open FPGA Stack (OFS) is a collection of RTL and open source software > +providing interfaces to access the instantiated RTL easily in an FPGA. > +OFS leverages the DFL for the implementation of the FPGA RTL design. > + > +OFS designs allow for the arrangement of software interfaces across > +multiple PCIe endpoints. Some of these interfaces may be PFs defined in > +the static region that connect to interfaces in an IP that is loaded via= Partial > Reconfiguration (PR). > +And some of these interfaces may be VFs defined in the PR region that > +can be reconfigured by the end-user. Furthermore, these PFs/VFs may use > +DFLs such that features may be discovered and accessed in user space > +(with the aid of a generic kernel driver like vfio-pci). The diagram > +below depicts an example design with two PFs and two VFs. In this > +example, it will export the management functions via PF0, > +PF1 will bind with virtio-net driver presenting itself as a network > +interface to the OS. The other functions, VF0 and VF1, leverage VFIO to > +export the MMIO space to an application or assign to a VM.:: > + > + +-----------------+ +--------------+ +-------------+ +----------= --+ > + | FPGA Management | | VirtIO | | User App | | Virtual = | > + | App | | App | | | | Machine = | > + +--------+--------+ +------+-------+ +------+------+ +-----+----= --+ > + | | | | > + +--------+--------+ +------+-------+ +------+------+ | > + | DFL Driver | |VirtIO driver | | VFIO | | > + +--------+--------+ +------+-------+ +------+------+ | > + | | | | > + | | | | > + +--------+--------+ +------+-------+ +------+------+ +----+----= --+ > + | PF0 | | PF1 | | PF0_VF0 | | PF0_VF1= | > + +-----------------+ +--------------+ +-------------+ +---------= --+ > + This diagram those Applications are DPDK application? > +As accelerators are specialized hardware, they are typically limited in > +the number installed in a given system. Many use cases require them to > +be shared across multiple software contexts or threads of software > +execution, either through partitioning of individual dedicated > +resources, or virtualization of shared resources. OFS provides several > +models to share the AFU resources via PR mechanism and hardware-based > virtualization schemes. > + > +1. Legacy model. > + With legacy model FPGA cards like Intel PAC N3000 or N5000, there is > + a notion that the boundary between the AFU and the shell is also the = unit of > + PR for those FPGA platforms. This model is only able to handle a > + single context, because it only has one PR engine, and one PR region = which > + has an associated Port device. > +2. Multiple VFs per PR slot. > + In this model, available AFU resources may allow instantiation of man= y VFs > + which have a dedicated PCIe function with their own dedicated MMIO sp= ace, > or > + partition a region of MMIO space on a single PCIe function. Intel PAC= N6000 > + card has implemented this model. > + In this model, the AFU/PR slot was not connected to port device. For = DFL's > view, > + the Next_AFU pointer in FIU feature header of port device points to N= ULL in > this > + model, so in AFU driver perspective, there is no AFU MMIO region mana= ged > by > + AFU driver. On the other hand, each VF can start with an AFU feature = header > without > + being connected to a FIU Port feature header. > + > +In multiple VFs per PR slot model, the port device can still be > +accessed using ioctls API which expose /dev/dfl-port.h device nodes, > +like port reset, get port info, whose APIs were mentioned in AFU > +section in this documentation. But it cannot access the AFU MMIO space > +via AFU ioctl APIs like DFL_FPGA_PORT_DMA_MAP because there is no AFU > +MMIO space managed in the AFU driver. Users can access the AFU resource > +by creating VF devices via PCIe SRIOV interface, and then access the VF = via > VFIO driver or assign the VF to VM. This section is not mentioned for DPDK? > + > +In multiple VFs per PR slot model, the steps to enable VFs are > +compatible with legacy mode which are mentioned in "FPGA virtualization > +- PCIe SRIOV" section in this documentation. The same of above. I think we can add a little description how to leverage the virtualization= for DPDK usage for PAC card. > + > +OFS provides the diversity for accessing the AFU resource to RTL develop= er. > +An IP designer may choose to add more than one PF for interfacing with > +IP on the FPGA and choose different model to access the AFU resource. > + > +There is one reference architecture design using the "Multiple VFs per P= R slot" > +model for OFS as illustrated below. In this reference design, it > +exports the FPGA management functions via PF0. PF1 will bind with > +virtio-net driver presenting itself as a network interface to the OS. > +PF2 will bind to the vfio-pci driver allowing the user space software > +to discover and interface with the specific workload like diagnostic > +test. To access the AFU resource, it uses SR-IOV to partition workload > interfaces across various VFs.:: > + > + +----------------------+ > + | PF/VF mux/demux | > + +--+--+-----+------+-+-+ > + | | | | | > + +------------------------+ | | | | > + PF0 | +---------+ +-+ | | > + +---+---+ | +---+----+ | | > + | DFH | | | DFH | | | > + +-------+ +-----+----+ +--------+ | | > + | FME | | VirtIO | | Test | | | > + +---+---+ +----------+ +--------+ | | > + | PF1 PF2 | | > + | | | > + | +----------+ | > + | | ++ > + | | | > + | | PF0_VF0 | PF0_VF1 > + | +-----------------+-----------+------------+ > + | | +-----+-----------+--------+ | > + | | | | | | | > + | | +------+ | +--+ -+ +--+---+ | | > + | | | Port | | | DFH | | DFH | | | > + +-----------+ +------+ | +-----+ +------+ | | > + | | | DEV | | DEV | | | > + | | +-----+ +------+ | | > + | | PR Slot | | > + | +--------------------------+ | > + | Port Gasket | > + +------------------------------------------+ > -- > 1.8.3.1