From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2A13942526; Wed, 6 Sep 2023 15:04:33 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 27F92402BE; Wed, 6 Sep 2023 15:04:33 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id B8F87402BE for ; Wed, 6 Sep 2023 15:04:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1694005471; x=1725541471; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=iKn/ptX7Mjb37EoR4nc1iFY6Cgdp36MxLTTkB2nBT3w=; b=Va2z/R48I7cMuHRrbHKlLsx2lX4G+IPSkk/ZIEOc0PJG1VjaTtACybcW 2wty5+C3hEab8w4JlAl4MYJ1riHe6siVmCVkVvADDwk66pWZSvpJUpVNB O1zhW8d7Kcyywb5OOOcU+K1dQr3LMEMP2ASw8BMeHot+FP1UWMUhg3KdI VHV0e6ldMS5Gz18hC5Vb5q158+wkV5Msm5Lx2caQ6Na1uvsSyDZ90LgMN j0QVXZbX7MI/vKUF95p1pdK24Gtx1WbsHW+WVCl/T3G8X1eFRk3NNWoWa NHWl1s/BpSjx4EML7GZZBzyoXjVfANy+5MK72KzSr9QsMUFufXKs7AbUs Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10825"; a="441034511" X-IronPort-AV: E=Sophos;i="6.02,232,1688454000"; d="scan'208";a="441034511" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Sep 2023 06:03:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10825"; a="691277603" X-IronPort-AV: E=Sophos;i="6.02,232,1688454000"; d="scan'208";a="691277603" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by orsmga003.jf.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 06 Sep 2023 06:03:25 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32; Wed, 6 Sep 2023 06:03:24 -0700 Received: from orsmsx602.amr.corp.intel.com (10.22.229.15) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 6 Sep 2023 06:03:24 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.32 via Frontend Transport; Wed, 6 Sep 2023 06:03:24 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.177) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.27; Wed, 6 Sep 2023 06:03:24 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NL5bNCPIjpLx3OzUvcJf44W9H7pGz0Fgo8JK0h58QtZPTM7RZImpJUfzBXVwZtgbSO6eMx22aXVhzla8gNpGEmJLD6RdershrQX8kr7kaw3JF/b6UloqXDe/W9q3LIXyVkMzMdW4AA44x3/Q0U8B7a6rwEqinjqu1tsHelbsmdMxsFdYjnZmjb4vzqSrm6WS+QXsL8gUU3FhYCs5b6MyFmfMlBNEHejU3PAMANs3NR/+pMV8sVGvoX4vhybS84QofXtudZwhVdffmSbH0clBvKx2z2Nziqbkezfx6Nk78AUliU2WJQLV/SKQD3zm0YZ27BlcP8iPPniVTdBf5cyvAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tKTO6OixyoGb2J4zMGeDlq/4KjMnUkGlayvnc8ZtnCU=; b=ZM+vBalpV+jN/IBdNpPorRL5ybkyyOcZBF2Xgmum1NyvYwLTEUhupXjePIgbgQrxJ824zynj3oFGJxgH0o/iBarhkethSZ0wepTdj0uj99iSENAmm9T2+TKr0kpmIKhPBpRZokUVk2JvWwlXQSe1Cg/t8OMdxYOV2vOKgXS9H5haqcT0yzaMryLXHCHn1p9UT+/K+DMCC9qdw5Dm0jSU3hcQqBlu+3EfELe1kbQ7l6HMwPWtPgJNpi12Su/PhSYaCp0x6FXSRov+3/AS4kb1v+mlLFbd+QLBO64vXykQ9oAzsxynqiko+xMLMlzxS+4xsXw0nwZaV9Mz07S2KirUIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from CH3PR11MB8362.namprd11.prod.outlook.com (2603:10b6:610:175::6) by PH7PR11MB8036.namprd11.prod.outlook.com (2603:10b6:510:248::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.33; Wed, 6 Sep 2023 13:03:21 +0000 Received: from CH3PR11MB8362.namprd11.prod.outlook.com ([fe80::88c2:332c:5e8a:25e]) by CH3PR11MB8362.namprd11.prod.outlook.com ([fe80::88c2:332c:5e8a:25e%7]) with mapi id 15.20.6745.030; Wed, 6 Sep 2023 13:03:19 +0000 From: "Xia, Chenbo" To: David Marchand , "dev@dpdk.org" CC: "thomas@monjalon.net" , "ferruh.yigit@amd.com" , "nipun.gupta@amd.com" , "Richardson, Bruce" , "Burakov, Anatoly" Subject: RE: [PATCH v2 03/15] bus/pci: rework MSIX discovery with VFIO Thread-Topic: [PATCH v2 03/15] bus/pci: rework MSIX discovery with VFIO Thread-Index: AQHZ1CO8AR/03XVakEmMWOiMyjbH5rAN3K6Q Date: Wed, 6 Sep 2023 13:03:19 +0000 Message-ID: References: <20230803075038.307012-1-david.marchand@redhat.com> <20230821113549.3191921-1-david.marchand@redhat.com> <20230821113549.3191921-4-david.marchand@redhat.com> In-Reply-To: <20230821113549.3191921-4-david.marchand@redhat.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CH3PR11MB8362:EE_|PH7PR11MB8036:EE_ x-ms-office365-filtering-correlation-id: 3f0c9b0a-32a0-4992-1a5a-08dbaed9a4be x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: kISz987qvfdETSgb4l2OVw7mYz+NkVzIKDk5gSLSz/Ou246LwIbl6He7QRiycJqGQLH8I/frym5qu7KzZP3ZLLpStYID58+7/apMEMO9iCFUKWnLlxk1Wv5HWqkPvkR8t0drUi4ZLAe4Vt7PTHdPO9xztsdMxqWrvvr8yVLaVM/Q7JrRGu4IwyQEq76kvkhtMPaugJaOBpkqbhy9cIbJCRzEQ4FlVLcfb21BIameu3xQ0bav3HBgbv56/tiM4Easst8ObSt5y82dJvsgeitEYR7u4UCVU80gSj9KIqotmPM6Ac3H+bbYjilWvkPHMl2HuXfagqv6T6NsueDuT8qoW0ogBOq3AgmzyVEYMBaLSxO+bZxw5U3GkekOo8k9icbOpD6QfFaIf1jDFQodn59akh8Z5awYXDwh/iv/n0oU6BT+E/y5kSA5LVNRbFmTtGqhh996kmob0vrI+uj6MqXoZnklf3sIBBt2K0c7Jc6RbJQ/AudAiN/AHAq7n45nTWEvrZZIXm4ZO+91T+4+ggQpuZ0bHg7J5Rox0TuJSC+S5ekiB0wQmZAjk110W9TMtC0ET8h/uMTR+W4zpHwsAWd7RLHJjPIwLHlkExelh2ii1D0Ho1GGByIRLuqqW5BtmLiK x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH3PR11MB8362.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(346002)(366004)(396003)(136003)(376002)(451199024)(186009)(1800799009)(8936002)(66476007)(66446008)(38100700002)(33656002)(38070700005)(86362001)(5660300002)(122000001)(82960400001)(2906002)(55016003)(8676002)(52536014)(478600001)(4326008)(41300700001)(83380400001)(7696005)(6506007)(316002)(53546011)(66946007)(107886003)(76116006)(66556008)(71200400001)(110136005)(9686003)(54906003)(64756008); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?4QKsHj65BaaTZFYofKa+iGhXNqahNyINwir9fGAfKRfrpBC6ENyoBj1Gr2MD?= =?us-ascii?Q?QQlpTUNQupBGsYOR483tr0MKv7GFhHs/eFZ5bUEQEhoL0A0rQ+8jtvYWriuO?= =?us-ascii?Q?9UmRTT/ZPWSvR5KZlxp8B+rAjPNMtXYmce506/WtE9bveb9VmAy+pUODWW7S?= =?us-ascii?Q?ZHk3dZ6l3IgFFcTyeJMMGZKxYDUbu+lLxHfpuOasBzik7s+p9VSuPNoxelt2?= =?us-ascii?Q?h11xAYZ2OFPsyeUZbjQm+kRaRoDPUyDUkagp95KnmhnKZiCX/sRevn+fVNtE?= =?us-ascii?Q?9kPvp5nflxKNRc06sWWYz+8vqMaSfGA9F6sUk4+EWbi4Tcvk4WKyhfEdNG0l?= =?us-ascii?Q?237Nqi6GCoQmOukN3KfCJkDbJIvY9zoUBbi6/F6WrmIqbmjEwgnsIMKxKc+v?= =?us-ascii?Q?ehVjaCtTy2oNT0Ar7KqNIjJQ0wBi82faInCaNeDfZ7ShRtk3wWEF/v8ghat5?= =?us-ascii?Q?idkKfY5YwVQWzwoQAKnwEwdnyuJEBC9mJ3bAZS7V6WJ9XTir0IjaLHm41UJV?= =?us-ascii?Q?I4OobTFzROxFY7v32AitJlS75kS1CB/6MvvdpTpWa2mRTmkundg9cbXHqA9y?= =?us-ascii?Q?k38mZcNvM8pYBFppMYWylFmOi3VF31XncCwC+rUuvgivOEXH1XDxPTwsL77e?= =?us-ascii?Q?wsMMrLjDDUXX1//L6ZR22gChH8aLbs1evp8LOXQyZSQAKF8zQ57+tvKXqITk?= =?us-ascii?Q?SHhumH55LgJpYjpdNrVuZFd4POyD8feB9/ZWwfuljmx+sIUAIUAok0gMaF82?= =?us-ascii?Q?jH2d0X0Op1THi9+pPc+xOXrb3fChYDYmvljlG8B+dHCYdhtYshE+8pz9uSlZ?= =?us-ascii?Q?o2vNbkhqb1hIUiJVSZ07p66IL3vonzwUuJgMqkva5cy+ViInZlu8OahekXHW?= =?us-ascii?Q?hCF4bvhtNctTCfx4ED3xo7IdO7JwavR8OcGmbIU0HOYX+G5ayNwTAAr9imXG?= =?us-ascii?Q?9ta9BJW7yysNieAPidXEOka8eJ3arI205RaqKDpirMnts5DGwQ5qjL+/a63j?= =?us-ascii?Q?U7bT3RzBtjXu/LMZ29eBQOd8HVjQ651T2vddZcuXb97T6Un22ixZoGF56j7y?= =?us-ascii?Q?pYICAlWsFTmTjd3TScp4uDEvKxfj13ujRcpc8yoyrpQ0nDVALH7YiwwhhD0Y?= =?us-ascii?Q?iHxwsf1FwnXazid6hzy5Ig88OqdfL5XjU50zhWT29hPMDm4YztIQpikQdE7u?= =?us-ascii?Q?WKpFDSA/gJbfus4RKDHrudoeqV1F6KhJFnm5zmxunRgePLZ8Y6UrB06YaBXP?= =?us-ascii?Q?+n0NhynGMFj1zuJXQgNIRgQj2Fu7McXm9dDw7uwXsLiX+NBpxiiP71kDgccS?= =?us-ascii?Q?W2YwK3GdeD+MWJGCF7CoEJ0w1d1wx3WPPBpE5OXSEOktOtUktbtTc/pt9Soi?= =?us-ascii?Q?ySctQKQ6aKRr1+/vmTXB+JSHf6jnASNwAjagMn3WuI6AkRAOsWOocN8x0Ra7?= =?us-ascii?Q?fPx803GeZcdnKk8itV+zhlIb7vhyDpQk3PT4EJWMorufu1Ozf2cglJah6qhA?= =?us-ascii?Q?CWOABZtw33/b05s54084rwOHMEjKcoIHZVdy/ORz9lsTVZbK5onKYbIl+A?= =?us-ascii?Q?=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH3PR11MB8362.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3f0c9b0a-32a0-4992-1a5a-08dbaed9a4be X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Sep 2023 13:03:19.7733 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PuDJJ9sunADDZY9AUxY75FexdZeDHMmza3nj3w0C9/PzuCW6fSzX4jad8q8FGaij6wg+FNdDKTz7snFaGCR/1g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB8036 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: David Marchand > Sent: Monday, August 21, 2023 7:36 PM > To: dev@dpdk.org > Cc: thomas@monjalon.net; ferruh.yigit@amd.com; Xia, Chenbo > ; nipun.gupta@amd.com; Richardson, Bruce > ; Burakov, Anatoly > Subject: [PATCH v2 03/15] bus/pci: rework MSIX discovery with VFIO >=20 > This is a preparatory step before using new helpers for finding PCI > capabilities. > In the code querying PCI capabilities for checking MSIX availability, > replace direct calls to VFIO fd with the existing helpers for reading > PCI configuration space: this requires setting VFIO fd in the PCI > device object than was done before this change and removes the need to > pass around this vfio_dev_fd variable. >=20 > Signed-off-by: David Marchand > --- > drivers/bus/pci/linux/pci_vfio.c | 27 ++++++++++----------------- > 1 file changed, 10 insertions(+), 17 deletions(-) >=20 > diff --git a/drivers/bus/pci/linux/pci_vfio.c > b/drivers/bus/pci/linux/pci_vfio.c > index 8fa7fa458f..958f8b3b52 100644 > --- a/drivers/bus/pci/linux/pci_vfio.c > +++ b/drivers/bus/pci/linux/pci_vfio.c > @@ -107,23 +107,16 @@ pci_vfio_write_config(const struct rte_pci_device > *dev, >=20 > /* get PCI BAR number where MSI-X interrupts are */ > static int > -pci_vfio_get_msix_bar(const struct rte_pci_device *dev, int fd, > +pci_vfio_get_msix_bar(const struct rte_pci_device *dev, > struct pci_msix_table *msix_table) > { > int ret; > uint32_t reg; > uint16_t flags; > uint8_t cap_id, cap_offset; > - uint64_t size, offset; > - > - if (pci_vfio_get_region(dev, VFIO_PCI_CONFIG_REGION_INDEX, > - &size, &offset) !=3D 0) { > - RTE_LOG(ERR, EAL, "Cannot get offset of CONFIG region.\n"); > - return -1; > - } >=20 > /* read PCI capability pointer from config space */ > - ret =3D pread64(fd, ®, sizeof(reg), offset + PCI_CAPABILITY_LIST); > + ret =3D rte_pci_read_config(dev, ®, sizeof(reg), > PCI_CAPABILITY_LIST); > if (ret !=3D sizeof(reg)) { > RTE_LOG(ERR, EAL, > "Cannot read capability pointer from PCI config > space!\n"); > @@ -136,7 +129,7 @@ pci_vfio_get_msix_bar(const struct rte_pci_device *de= v, > int fd, > while (cap_offset) { >=20 > /* read PCI capability ID */ > - ret =3D pread64(fd, ®, sizeof(reg), offset + cap_offset); > + ret =3D rte_pci_read_config(dev, ®, sizeof(reg), cap_offset); > if (ret !=3D sizeof(reg)) { > RTE_LOG(ERR, EAL, > "Cannot read capability ID from PCI config > space!\n"); > @@ -148,7 +141,7 @@ pci_vfio_get_msix_bar(const struct rte_pci_device *de= v, > int fd, >=20 > /* if we haven't reached MSI-X, check next capability */ > if (cap_id !=3D PCI_CAP_ID_MSIX) { > - ret =3D pread64(fd, ®, sizeof(reg), offset + > cap_offset); > + ret =3D rte_pci_read_config(dev, ®, sizeof(reg), > cap_offset); > if (ret !=3D sizeof(reg)) { > RTE_LOG(ERR, EAL, > "Cannot read capability pointer from PCI > config space!\n"); > @@ -163,14 +156,14 @@ pci_vfio_get_msix_bar(const struct rte_pci_device > *dev, int fd, > /* else, read table offset */ > else { > /* table offset resides in the next 4 bytes */ > - ret =3D pread64(fd, ®, sizeof(reg), offset + cap_offset > + 4); > + ret =3D rte_pci_read_config(dev, ®, sizeof(reg), > cap_offset + 4); > if (ret !=3D sizeof(reg)) { > RTE_LOG(ERR, EAL, > "Cannot read table offset from PCI config > space!\n"); > return -1; > } >=20 > - ret =3D pread64(fd, &flags, sizeof(flags), offset + > cap_offset + 2); > + ret =3D rte_pci_read_config(dev, &flags, sizeof(flags), > cap_offset + 2); > if (ret !=3D sizeof(flags)) { > RTE_LOG(ERR, EAL, > "Cannot read table flags from PCI config > space!\n"); > @@ -306,9 +299,6 @@ pci_vfio_setup_interrupts(struct rte_pci_device *dev, > int vfio_dev_fd) > if (rte_intr_fd_set(dev->intr_handle, fd)) > return -1; >=20 > - if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd)) > - return -1; > - > switch (i) { > case VFIO_PCI_MSIX_IRQ_INDEX: > intr_mode =3D RTE_INTR_MODE_MSIX; > @@ -838,6 +828,9 @@ pci_vfio_map_resource_primary(struct rte_pci_device > *dev) > if (ret) > return ret; >=20 > + if (rte_intr_dev_fd_set(dev->intr_handle, vfio_dev_fd)) > + goto err_vfio_dev_fd; > + > /* allocate vfio_res and get region info */ > vfio_res =3D rte_zmalloc("VFIO_RES", sizeof(*vfio_res), 0); > if (vfio_res =3D=3D NULL) { > @@ -869,7 +862,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device > *dev) > /* get MSI-X BAR, if any (we have to know where it is because we > can't > * easily mmap it when using VFIO) > */ > - ret =3D pci_vfio_get_msix_bar(dev, vfio_dev_fd, &vfio_res->msix_table); > + ret =3D pci_vfio_get_msix_bar(dev, &vfio_res->msix_table); > if (ret < 0) { > RTE_LOG(ERR, EAL, "%s cannot get MSI-X BAR number!\n", > pci_addr); > -- > 2.41.0 Reviewed-by: Chenbo Xia