From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E39EA0471 for ; Tue, 16 Jul 2019 08:47:20 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3681C3423; Tue, 16 Jul 2019 08:47:20 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 04A213423 for ; Tue, 16 Jul 2019 08:47:18 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x6G6jWwY003414; Mon, 15 Jul 2019 23:47:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=WB98skcEaprLKqjhj86RF1UT5B2McBkzzpbAgHks7VA=; b=vhM7zhhCcKSyVYh2o38r9OhjIHWnuIdhwjHKVQy/5uojd3OfkHA3DdpmZB/DCKZWvcHY g14joW20EM1UQisLEfg827HxQdLSYHAnDnNCJjhWgDxD4qjB7zXFEEsGMclK8HmtW+ip 0B2FHA9XwA3LYnujA5IQ6Nrs4p2DzgXCECli8rH9MxuO40yLHJ1qZfroYEZ7mYVm/9Am giFgUyFEfFL7tp4XNlAShY1qLgRj3bacMfF5v1BXuEeyeWM3RFoyLmEY/a45UuXDA6Z9 2AqyH4yFN6H2yQjY7sbo6CF26wbHX9dnPcONpUi9ImJB0seZI8XO/LzL8qmx2NCRTg1s BQ== Received: from sc-exch01.marvell.com ([199.233.58.181]) by mx0b-0016f401.pphosted.com with ESMTP id 2ts0a21use-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 15 Jul 2019 23:47:17 -0700 Received: from SC-EXCH02.marvell.com (10.93.176.82) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Mon, 15 Jul 2019 23:47:14 -0700 Received: from NAM03-DM3-obe.outbound.protection.outlook.com (104.47.41.51) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Mon, 15 Jul 2019 23:47:14 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KgKfcY+5RiDq6I13p0uAkDdBPaWbIfWgyV0br6Qw/PWdkiVX8Ttot1tGPYZEdxzR2R3iJW9ECUx6/+VOyardM2xUa+mksi5VSpSTSfyiMDiumLVSFE/nYtHOa9Ty7O2eSfE4P2W+dHyb/2vmZTvUGpXtaN1zqbo7Lg9y3M1qGyNrNwbhC25tn55jBmtKrJaZxAUbW0LcLuxI8S179ZhSnbZ45zxdzqHj/tmodVlKZGSb2YthTSVSEzvc1UdAGGygKNp+K0yxVDkF/cuhq09sZRWnI7U6SE/wpDfiNjIdd0BgKNVSbxpzZ8/22tHM79kSy7Vy1gtjEqKpRJl6LfGttA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WB98skcEaprLKqjhj86RF1UT5B2McBkzzpbAgHks7VA=; b=IsXbv4aiwCpk4WS+j1WLMGaNHJsXgFVOno39Cm16pnHFV8xSmHJTJS+94AKF+cPMpzRp7piOLO+jqo18cvyFYajA5FrXvSjHqaSiu4M2ydpAIUGKLd+j8klCdh77XSuPhfqv5wwlcNJlQVa/1An2GcVUxKXnBrcayvJtWN7GMYpRArL3qoPJs0nQUbxIRq2q8ut10oq9OKqPseSoCky7FXDiBASx0OPTUI0y9ELSryY0zt++1ySPThK+Mg7y7WeMwF5ECd3vs+FwEknE+Rhr5cjq0Dcm3eHNj7Oxhw1cwPXFyLZJT41cQAtioa9Z6CI9SWsORYYhyiYk+He2kviG4A== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=marvell.com;dmarc=pass action=none header.from=marvell.com;dkim=pass header.d=marvell.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WB98skcEaprLKqjhj86RF1UT5B2McBkzzpbAgHks7VA=; b=QQj997xcNwnfm6jab7ieuVtmQIoytLJtMq3blUiNKzOS0xC8CCxKlGPrEk4o+Z0naoAWhDyYbC1+ZOaoBbmB8G70WP+RFBoWav3dTFgzBDYmCDu45+pG947NBDWH+G8htb5iT1fd+9OsSKIdbX4CGhXqNrLMUqxNeSoQq0HzVRw= Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by BYAPR18MB2360.namprd18.prod.outlook.com (20.179.90.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2073.10; Tue, 16 Jul 2019 06:47:10 +0000 Received: from BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862]) by BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862%4]) with mapi id 15.20.2073.012; Tue, 16 Jul 2019 06:47:10 +0000 From: Jerin Jacob Kollanukkaran To: "Hyong Youb Kim (hyonkim)" , David Marchand , Thomas Monjalon , "Ferruh Yigit" , Alejandro Lucero , Anatoly Burakov CC: "dev@dpdk.org" , "John Daley (johndale)" , Shahed Shaikh , "Nithin Kumar Dabilpuram" Thread-Topic: [RFC PATCH] vfio: avoid re-installing irq handler Thread-Index: AdU7LOBENm/7fWlUSVuvO2rop1WEPwAbdfHgAAD0wuA= Date: Tue, 16 Jul 2019 06:47:09 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [106.200.248.176] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 88c37515-c3c2-4ae4-dab6-08d709b96d38 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:BYAPR18MB2360; x-ms-traffictypediagnostic: BYAPR18MB2360: x-ms-exchange-purlcount: 1 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 0100732B76 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(346002)(366004)(396003)(136003)(39860400002)(376002)(199004)(189003)(13464003)(7736002)(229853002)(68736007)(4326008)(305945005)(66556008)(99286004)(5660300002)(66446008)(64756008)(52536014)(2906002)(316002)(53936002)(53546011)(6506007)(6246003)(7696005)(76176011)(107886003)(478600001)(86362001)(8936002)(66476007)(3846002)(25786009)(66946007)(71200400001)(71190400001)(74316002)(966005)(33656002)(6116002)(66066001)(14454004)(81166006)(81156014)(110136005)(186003)(11346002)(446003)(26005)(9686003)(54906003)(476003)(256004)(6436002)(8676002)(6306002)(102836004)(486006)(55016002)(76116006); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB2360; H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: l6pFAMuWGfOJe5/v8ymFkDi3QuR0RgSkXrCD2VtxbHNlSHNZkQN3kZKLQfpn5l713epbvf6RV4Gx76x/QAkgkouRS8W6vsM9bNaFQRFJ8nKO6n4HwamfMWjdVkGcXsDDctFUuZ4bW3myADRI6DUEA1vCxuCFA+/+CDzQ3NYlx4VjsOItcpqLBo4pKG6+V2xdEdDEUCkNhEpM6/XPEnU7dUFBCqP0ffR6dnXrWsORAVIIsfUyyaqvPh4SDxyuGbtpMoudqkfxniLOSJuDDzzOkcWuVG4lfxEeyBftXPFHrxvaWh/2KkmtV/9lRWxJK54YJlSb92IfXsIoNHVo/lKlLSdlDmAU2LYnEjrK08OdZhdybGs16MJ5kQPgqwXGNH3EZlAFyhATiHyFzVIZsC1uQQlWjJ/90LwtRPdOZh+7fwY= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 88c37515-c3c2-4ae4-dab6-08d709b96d38 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jul 2019 06:47:09.9788 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jerinj@marvell.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2360 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-16_02:2019-07-16,2019-07-16 signatures=0 Subject: Re: [dpdk-dev] [RFC PATCH] vfio: avoid re-installing irq handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Hyong Youb Kim (hyonkim) > Sent: Tuesday, July 16, 2019 11:28 AM > To: Jerin Jacob Kollanukkaran ; David Marchand > ; Thomas Monjalon > ; Ferruh Yigit ; Alejandro > Lucero ; Anatoly Burakov > > Cc: dev@dpdk.org; John Daley (johndale) ; Shahed > Shaikh ; Nithin Kumar Dabilpuram > > Subject: RE: [RFC PATCH] vfio: avoid re-installing irq handler >=20 > > > A rough patch for the approach mentioned earlier. It is only for > discussion. > > > http://mails.dpdk.org/archives/dev/2019-July/138113.html > > > > > > To try this out, first revert the following then apply. > > > commit 89aac60e0be9 ("vfio: fix interrupts race condition") > > > > Yes. This patch has to be to reverted. It changes the existing > > interrupt behavior and does not address the MSIX case as well. > > > > I think, The clean fix would be to introduce rte_intr_mask() and > > rte_intr_unmask() by abstracting the INTX and MSIX differences And let > > qede driver call it as needed. > > > > Thoughts? >=20 > Hi, Hi Hyong, >=20 > You are proposing these? > - Add rte_intr_mask_intx, rte_intr_unmask_intx. > No APIs for masking MSI/MSI-X as vfio-pci does not support that. > - Modify PMD irq handlers to use rte_intr_unmask_intx as necessary. No, introduce the rte_intr_mask() and rte_intr_unmask(). For MSIX + Linux VFIO, That API can return -ENOSUP as Linux VFIO+MSIX is no= t supporting. Another platform/eal may support it. Mask and unmask is operation is known to all IRQ controllers. So, IMO, As far as abstraction is concerned it will be good fit. > That might be too intrusive. And too much work for the sake of INTx.. > Anyone really using/needing INTx these days? :-) Yup. Mask needs to called only for only qede INTx. Looks like qede Has MSIX and INTX separate handler. So this mask can go to qede INTx >=20 > The following drivers call rte_intr_enable from their irq handlers. So wi= th > explicit rte_intr_unmask_intx, all these would need to do "if using intx, > unmask"? >=20 > atlantic, avp, axgbe, bnx2x, e1000, fm10k, ice, ixgbe, nfp, qede, sfc, > vmxnet3 No change on these PMDs. > And nfp seems to rely on rte_intr_enable to re-install irq handler to unm= ask > a vector in MSI-X Table? >=20 > if (hw->ctrl & NFP_NET_CFG_CTRL_MSIXAUTO) { > /* If MSI-X auto-masking is used, clear the entry */ > rte_wmb(); > rte_intr_enable(&pci_dev->intr_handle); >=20 > With David's patch and mine, this handler would have to first > rte_intr_disable() and then enable, if such unmasking is really necessary= .. >=20 > As for the semantics of rte_intr_enable/disable, I am ok as is. > - "enable": put things in a state where NIC can send an interrupt, and > PMD/app gets a callback. > Whether this involves unmasking for INTx is hidden. > - "disable": put things in a state where NIC cannot send an interrupt. It looks OK to me. My only thought was, Since mask and unmask is a common irq controller operation. We may not need to add A lot of common code(Introducing a state) to hide unmask INTx. More over as you said, There is may only handful of devices uses INTX. IMO, mask and unmask API is good fit as eal abstraction. But Using a separate API or hide inside eal to solve this problem is good q= uestion. May be more thoughts from another quys will be good. We will try to send a version with mask/unmask API to see the changes requi= red. >=20 > Regardless of vfio changes, we should probably remove rte_intr_enable > from qede_interrupt_handler (the MSI/MSI-X interrupt handler), to make > usage/intention clear.. Yes. Anyway this change is required. >=20 > Thanks. > -Hyong