From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9C222A0557; Thu, 26 May 2022 08:48:08 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8CA274161A; Thu, 26 May 2022 08:48:08 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mails.dpdk.org (Postfix) with ESMTP id DC5E640DF7; Thu, 26 May 2022 08:48:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653547686; x=1685083686; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=A/p/jRE72STzv6MCyRy2UlMF7pgIATIUsjsCCVxHMVg=; b=Qst9qOEgHPAKGsdrgdbWsdUPWwGFo+ueyIKz6g10ywj2ONiQplXQGG74 G/GwDwvaMWxhBskPd/vLdM0nfjqs0ZsKJ3leNi257dE0obdE+wAsxblSU YtcMvwvr5zrW1Fx6R2UKKjtuSxmW0hxWNgc72OVL9RY4MoqJz1fffJGEX T1rcyiqshF76O8jJ0wwWHTY4mda7BATNNnEZoLKGCBDCr9mXygyQKhdTs FXbyT+uW1WBMYzwFNfEaRkCP6XMhxGQmT6Pd1Mb6THVcMMoPBNAq16TC0 s+ARKq3+9hJv9hELZUCIFTVVFZ/vbthDNAK5karYoty44TwoJL+zZ/mlA Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10358"; a="271628201" X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="271628201" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 May 2022 23:47:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,252,1647327600"; d="scan'208";a="704379996" Received: from fmsmsx606.amr.corp.intel.com ([10.18.126.86]) by orsmga004.jf.intel.com with ESMTP; 25 May 2022 23:47:56 -0700 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx606.amr.corp.intel.com (10.18.126.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Wed, 25 May 2022 23:47:55 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Wed, 25 May 2022 23:47:55 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.47) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Wed, 25 May 2022 23:47:55 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Km4FIZfP+zh16RnPFM+Wr0kP6vFya9V46sluKM6OCU1ZClVTtDis81dqxsqQfQHoQOA1r3VvyVj2fclPKkMRSbSu5vzD/rgCO24tNZ40jtzxb2y13WMCbKk9Jq3nVMSbFWPIHmDcAkEhwFM39kWpOMW6KjLnCbDECrd40KP+AwiCVhdKD8gVAPipQ2MBX7JfRJ3YOGTb2ajVdElCpXyzHEZRr7/NMdcU61324pC5vdEo1zidCcJoeugkNmbnEgYm5Jc529AX0WM/L050DwjIXEElE087JLNC4OQjSfb19vAQR5Fv495Ew5IUWFE2eifGxa7K2b7L7QhV0RYTO+1Iog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oeKgH3eR1cgfNh85m0oP9Gm9QxVjaHMJ1170BllOXyY=; b=LAvG2bupTTlyDVrCdEkbsCDpr7XWN7wGbxjl1mogjLobLH4OTl7ncVzxBe1mWvphFC8s9LXHkZaTTbcHt1iPJPiSAOcf9P5wfTTN54mHfYKm/0hxzFdNEYyVRzy5n2Frr6bWofhmdBMOqhPx1Yfr/03ET0GTyU74RUxZZhvBr+Nsq70wV7j6OwMF8bptBSH0QuEf+Ent3NPlDYgwZOtRNq58w0U7AQcvu8y13iWW5t7p1BNaViagKZSVafc+c65JkiRbYaKeeNLOREY1q0z4hrT7U0S+YGu6E4XFhFrh8wpg4ETpGNuZ5Omn93pPpPeAy1Ji+jhhtxDR6MfEzOU16Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DM6PR11MB4252.namprd11.prod.outlook.com (2603:10b6:5:201::26) by BL3PR11MB6529.namprd11.prod.outlook.com (2603:10b6:208:38c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.22; Thu, 26 May 2022 06:47:54 +0000 Received: from DM6PR11MB4252.namprd11.prod.outlook.com ([fe80::b9d5:600c:9f13:bc96]) by DM6PR11MB4252.namprd11.prod.outlook.com ([fe80::b9d5:600c:9f13:bc96%7]) with mapi id 15.20.5293.013; Thu, 26 May 2022 06:47:54 +0000 From: "Xu, Rosen" To: "Huang, Wei" , "dev@dpdk.org" , "thomas@monjalon.net" , "nipun.gupta@nxp.com" , "hemant.agrawal@nxp.com" CC: "stable@dpdk.org" , "Zhang, Tianfei" , "Zhang, Qi Z" Subject: RE: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Topic: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga doc Thread-Index: AQHYcLBXkS5NfxLgPk6vYZcalaKsUq0wt5uw Date: Thu, 26 May 2022 06:47:54 +0000 Message-ID: References: <1652862549-13131-1-git-send-email-wei.huang@intel.com> <1653535974-1379-1-git-send-email-wei.huang@intel.com> <1653535974-1379-6-git-send-email-wei.huang@intel.com> In-Reply-To: <1653535974-1379-6-git-send-email-wei.huang@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOTcyNmVmYjAtODc4Yi00NTljLWFjODEtNDFhZTRhZjhjNjQwIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiMGVjYWluSjVPd0grUzFta3hFTEI1QXgwYnJ1cTB2WXpjTTZZUlcwSU1hVUE4VjVWeGd2VHdlanpaZVZ3QkZMQyJ9 x-ctpclassification: CTP_NT authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 71ecea28-83be-4a73-7b5f-08da3ee3a911 x-ms-traffictypediagnostic: BL3PR11MB6529:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: LF+ZbKvoewVEBI3SuX9uEUUShrfNXETe7Vj1gdJC15l7Jinqd2ALW3QV94sexxJaojHLB9koSbQtFSIHbVTZl744ForEKMavTJYCaUCqUessPusudhx8NYKUxNXtZHXMJeEYaKY2/s8iJyui0OuCEX4WHlpOeHqjI0DS4T3MlwmhrheH4QalOEkHXXTtg5LM2uXYjOPLiBm+tjX6MJd29h5SkMEca+kPwtuGmSBKURa3vcawoHT2mYYLobh74Ve5foZBNF1Jecsj56rOyfGkBLhtCVovhXefVvoJZuQ9LFmuc6iR/tTVzzmt18wIWltbrXSaqpGQBJaxWitXdYPyHCcfEfS0muVLJXbGzTqFCJm8CkkHisLpBFZBbVkfXmLYecgEVduarVVXzEJRQ4PJqbZwXT/pS2xjLODkoKI6CidaD9GbiRVddtbarL7yIQT0FB4mRWWO5S/sH19iZMwJvN0hfsRVFODzNIJinzwaM0bg8CfY2GGtwv/90u4ae3yYHSXRUUXUJhNhVCNJSA1vvToKLRSuegaBRnnX9395yz/BU+xH98Ugu4v8+1z7Szq1RY/DqPKYM4UVR93MKu/+Zmh1afvZvO6Ex1orOJY7+1pPuKs4ocMVX0oHyDT+ypApmG0BsvlvnYAoBjHOSCAd75atXw2mU9mZ0OabCH7E1stJ68kDIAovsiggM7dP/lVtsY7pfzQDZ/5Vx+xLOEULUQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB4252.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(64756008)(76116006)(66446008)(66476007)(107886003)(71200400001)(86362001)(122000001)(186003)(83380400001)(8676002)(55016003)(8936002)(82960400001)(110136005)(54906003)(26005)(53546011)(9686003)(7696005)(6506007)(316002)(38070700005)(38100700002)(66556008)(5660300002)(2906002)(33656002)(66946007)(4326008)(508600001)(52536014); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?lJzZkB5tW9znXkhqrDwHpWOkTNSsWVFSRJQswoEzhYL+8un2ej80aUkvS8uq?= =?us-ascii?Q?TePUEvw5Wqn7aThzTAMhb7FZB1eZ0cHCl1BwFCNSQcrlCnLSK3pfetEtpCLT?= =?us-ascii?Q?uC4dHDgGtXkfIrCIN7AbFYNJD/6tcaFzymy49RrRZeoKBFc3ZCXrEPvkAoMj?= =?us-ascii?Q?l56Eb2QQ6W1SIv1Vx5XDAMra5W2OMghF1PGoAFDcbufjYPnK/POD6KmJhxOq?= =?us-ascii?Q?Jj7KWVPLn2pLREmt6/8jloVo+a3jLfR+Y9VeXT8XzhZyWDQtIUoyHQzoUURj?= =?us-ascii?Q?nGrlC4sVMgSdLlDb4UPI1w1aleaDcxa/CIAtX3IZ0uWmXadnCyS8/D2xoeC3?= =?us-ascii?Q?C8BqAW8xFnn38Rmhcgp5+yYQF97so5Kmk9lxsQXLs1CTNTfyC9ex+Lq5wcNS?= =?us-ascii?Q?9RhNtxc0PyzddJ8Ix6tBWkP/lLkFglQb7oFigZ268i56y3ByTHRxu5Hi2A8F?= =?us-ascii?Q?h9/IqNHO8C5n4qbMLMK8U42pi6KoIudjYFTaEGOY+8/EDOIzuVFhcLTmpsY0?= =?us-ascii?Q?kC8zmgi4ITD7pMA8syNTaqhb4Hu5Sk6SWC/7B4zrmA9qhIvT/QuABUvTRzPH?= =?us-ascii?Q?Yw0HBhDgFzfQV9q26Y62UMZFWqcGbaIXU20ReMhBkEvJmbtRzdThTZUi3Ivb?= =?us-ascii?Q?GthgbRJcfNkFhlCJVS30xuxnbboLE0g7zHzjflPh8m2IfYDGXGrtJ5mJAl8Z?= =?us-ascii?Q?S7AyA+pr62GYeDKrSAlMwarqjSImAw9WOfu0S+gi5ORSveu8dFd6qabMU7ch?= =?us-ascii?Q?w76f2JPuseFIzgoq3k4WOMLjXCCSiSVR/6e+RigSJHb4LbAUJtJ05QuApizs?= =?us-ascii?Q?w/NGBjLG2cKv5szjKm1LFP8nUojWN/GcC5Y18fuNRuTGIblixJDC3fx9xuzr?= =?us-ascii?Q?H7Hj88yZv1E+sFtTQ6Ugpdh4CZNpLWatHxMn4N6RqlSyl/DbuL6sMUyULO0a?= =?us-ascii?Q?yQhasv169r0yzQRnRDz8fRI+0BT19WssJGL2lGImp055kPHBHqFR2vA2uEhX?= =?us-ascii?Q?VwbBP0Rqpa6tKTiFTG4pZw6912t9i7VOmfZ/YPX9P+DlxyB7hZBHIBiuR5vv?= =?us-ascii?Q?9Ed2yFwBEIxLI61KzahEU4QGff7zQQA4ILLiLDN4KKeGg/WxLclc0t1D6OUd?= =?us-ascii?Q?JSWeIjgFGVLdiwAsY1gUS1dhq9pYwK+n4C1M0W8A7XV8SQQGUFyo3Tv9sbNq?= =?us-ascii?Q?Ape1MbsiPOpmnkI+Tn9Lc0XXsHQgwyvmpCfJ8tOr45I8vi5T8lXbGkfBUofr?= =?us-ascii?Q?ytBgEqwPt54AQFo2Kl/akE++bzIZh4eUNaCDMlj4sSXLPzAu8+dB8gONW1q2?= =?us-ascii?Q?9nD6VaCNrxX5xxH6z084fBZXCXAz7rO+zTEmzJqFM2k05SSt7DLaYAk5Aqi8?= =?us-ascii?Q?0fZUlo1Kdzv3ytTkOa7v0C/O+eR9dMwmZahDtYcUih+GbiJ1KLXWZu+edFN+?= =?us-ascii?Q?rq5EtnhmoYMNQfeSr4CXNIw1yboAos6F/HJ3vLaD6SG1Go2l2bTEzu8s1eEZ?= =?us-ascii?Q?kE3QUtAZyGv+aM8kL/xsG4VwJ3maDsyQlIbY1QOdbqCfPHkdCvsU8SxRqjBI?= =?us-ascii?Q?Mzg7ISr82T9+3xNGlKYwciPAA5unpvH7/k1X19QgV4o1lnMFUdXY/vVsWTHF?= =?us-ascii?Q?0lrIemPQSF9Ua4ouSNFpB2qF0oXTGkyWXExZxTDzwYpQ6GGn/CJMnYxjwAte?= =?us-ascii?Q?6Gkt8f8fi4UZqD+sYey5Yu1iH4CzjQvujazRJ7mWLufYVb/V2w34FbUGjF6K?= =?us-ascii?Q?m+HSVML0dw=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4252.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 71ecea28-83be-4a73-7b5f-08da3ee3a911 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 May 2022 06:47:54.1027 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: BDqCbG++ujqYNKlQLmCHaXBVZrGfsc05u1ihOumIniFTqXKnTx0N7frfh8ysoU4KiDQEgNtkzLrffTK/3RaCmA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6529 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Wei, > -----Original Message----- > From: Huang, Wei > Sent: Thursday, May 26, 2022 11:33 > To: dev@dpdk.org; thomas@monjalon.net; nipun.gupta@nxp.com; > hemant.agrawal@nxp.com > Cc: stable@dpdk.org; Xu, Rosen ; Zhang, Tianfei > ; Zhang, Qi Z ; Huang, Wei > > Subject: [PATCH v3 5/5] guides/rawdevs: add description of ofs in ifpga d= oc >=20 > OFS (Open FPGA Stack) specification is introduced briefly. >=20 > Signed-off-by: Wei Huang > --- > doc/guides/rawdevs/ifpga.rst | 114 > ++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 113 insertions(+), 1 deletion(-) >=20 > diff --git a/doc/guides/rawdevs/ifpga.rst b/doc/guides/rawdevs/ifpga.rst > index dbd0d6e..8809bfc 100644 > --- a/doc/guides/rawdevs/ifpga.rst > +++ b/doc/guides/rawdevs/ifpga.rst > @@ -1,5 +1,5 @@ > .. SPDX-License-Identifier: BSD-3-Clause > - Copyright(c) 2018 Intel Corporation. > + Copyright(c) 2018-2022 Intel Corporation. >=20 > IFPGA Rawdev Driver > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > @@ -100,3 +100,115 @@ The following device parameters are supported: >=20 > If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and > identifies AFU Bit Stream file. > + > + > +Open FPGA Stack > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +Open FPGA Stack (OFS) is a collection of RTL and open source software > +providing interfaces to access the instantiated RTL easily in an FPGA. > +OFS leverages the DFL for the implementation of the FPGA RTL design. > + > +OFS designs allow for the arrangement of software interfaces across > +multiple PCIe endpoints. Some of these interfaces may be PFs defined in > +the static region that connect to interfaces in an IP that is loaded via= Partial > Reconfiguration (PR). > +And some of these interfaces may be VFs defined in the PR region that > +can be reconfigured by the end-user. Furthermore, these PFs/VFs may use > +DFLs such that features may be discovered and accessed in user space > +(with the aid of a generic kernel driver like vfio-pci). The diagram > +below depicts an example design with two PFs and two VFs. In this > +example, it will export the management functions via PF0, > +PF1 will bind with virtio-net driver presenting itself as a network > +interface to the OS. The other functions, VF0 and VF1, leverage VFIO to > +export the MMIO space to an application or assign to a VM.:: > + > + +-----------------+ +--------------+ +-------------+ +----------= --+ > + | FPGA Management | | VirtIO | | User App | | Virtual = | > + | App | | App | | | | Machine = | > + +--------+--------+ +------+-------+ +------+------+ +-----+----= --+ > + | | | | > + +--------+--------+ +------+-------+ +------+------+ | > + | DFL Driver | |VirtIO driver | | VFIO | | > + +--------+--------+ +------+-------+ +------+------+ | > + | | | | > + | | | | > + +--------+--------+ +------+-------+ +------+------+ +----+----= --+ > + | PF0 | | PF1 | | PF0_VF0 | | PF0_VF1= | > + +-----------------+ +--------------+ +-------------+ +---------= --+ > + > +As accelerators are specialized hardware, they are typically limited in > +the number installed in a given system. Many use cases require them to > +be shared across multiple software contexts or threads of software > +execution, either through partitioning of individual dedicated > +resources, or virtualization of shared resources. OFS provides several > +models to share the AFU resources via PR mechanism and hardware-based > virtualization schemes. > + > +1. Legacy model. > + With legacy model FPGA cards like Intel PAC N3000 or N5000, there is > + a notion that the boundary between the AFU and the shell is also the = unit > of > + PR for those FPGA platforms. This model is only able to handle a > + single context, because it only has one PR engine, and one PR region > which > + has an associated Port device. > +2. Multiple VFs per PR slot. > + In this model, available AFU resources may allow instantiation of man= y VFs > + which have a dedicated PCIe function with their own dedicated MMIO > space, or > + partition a region of MMIO space on a single PCIe function. Intel PAC > N6000 > + card has implemented this model. > + In this model, the AFU/PR slot was not connected to port device. For = DFL's > view, > + the Next_AFU pointer in FIU feature header of port device points to N= ULL > in this > + model, so in AFU driver perspective, there is no AFU MMIO region > managed by > + AFU driver. On the other hand, each VF can start with an AFU feature > header without > + being connected to a FIU Port feature header. > + > +In multiple VFs per PR slot model, the port device can still be > +accessed using ioctls API which expose /dev/dfl-port.h device nodes, > +like port reset, get port info, whose APIs were mentioned in AFU > +section in this documentation. But it cannot access the AFU MMIO space > +via AFU ioctl APIs like DFL_FPGA_PORT_DMA_MAP because there is no > AFU > +MMIO space managed in the AFU driver. Users can access the AFU resource > +by creating VF devices via PCIe SRIOV interface, and then access the VF = via > VFIO driver or assign the VF to VM. > + > +In multiple VFs per PR slot model, the steps to enable VFs are > +compatible with legacy mode which are mentioned in "FPGA virtualization > +- PCIe SRIOV" section in this documentation. > + > +OFS provides the diversity for accessing the AFU resource to RTL develop= er. > +An IP designer may choose to add more than one PF for interfacing with > +IP on the FPGA and choose different model to access the AFU resource. > + > +There is one reference architecture design using the "Multiple VFs per P= R > slot" > +model for OFS as illustrated below. In this reference design, it > +exports the FPGA management functions via PF0. PF1 will bind with > +virtio-net driver presenting itself as a network interface to the OS. > +PF2 will bind to the vfio-pci driver allowing the user space software > +to discover and interface with the specific workload like diagnostic > +test. To access the AFU resource, it uses SR-IOV to partition workload > interfaces across various VFs.:: > + > + +----------------------+ > + | PF/VF mux/demux | > + +--+--+-----+------+-+-+ > + | | | | | > + +------------------------+ | | | | > + PF0 | +---------+ +-+ | | > + +---+---+ | +---+----+ | | > + | DFH | | | DFH | | | > + +-------+ +-----+----+ +--------+ | | > + | FME | | VirtIO | | Test | | | > + +---+---+ +----------+ +--------+ | | > + | PF1 PF2 | | > + | | | > + | +----------+ | > + | | ++ > + | | | > + | | PF0_VF0 | PF0_VF1 > + | +-----------------+-----------+------------+ > + | | +-----+-----------+--------+ | > + | | | | | | | > + | | +------+ | +--+ -+ +--+---+ | | > + | | | Port | | | DFH | | DFH | | | > + +-----------+ +------+ | +-----+ +------+ | | > + | | | DEV | | DEV | | | > + | | +-----+ +------+ | | > + | | PR Slot | | > + | +--------------------------+ | > + | Port Gasket | > + +------------------------------------------+ > -- > 1.8.3.1 Reviewed-by Rosen Xu