From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 32C21A0471 for ; Mon, 15 Jul 2019 07:35:31 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 44935325F; Mon, 15 Jul 2019 07:35:30 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id B85E13256; Mon, 15 Jul 2019 07:35:27 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x6F5ZK4m014239; Sun, 14 Jul 2019 22:35:20 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=; b=Pia1qytPnO6EGjKo4iurnx1U89myiOuHgpzXgMjHe0JXh+RntbE8y5dQAuFJXib0+RJW NRvoAwxn9S/m6v7P0ozcLbCKHoIAdwVx29Bzjfjs2D7fNID+kluZ499b9CCgm0m+BLn5 kVu4JqjA5cfuPbwP9NO+W9DZLCfjBytm8a89Td6mo20DfxEQrNKZA/da5sQ5NAIW8w0k 8PpsyuMphQlO/kQYHwojV4cuygw4GOZ8DgdT71nY4y/YFifqoXkcKAgFubgQ/SGPTDt4 UldbMHJsTDigX+3VhYlikAA6vbL31Vnj58bCHIISF4nB/O2D1yeyq8hT/NA24sMqX4cP RA== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0a-0016f401.pphosted.com with ESMTP id 2tqcnpp71k-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Sun, 14 Jul 2019 22:35:20 -0700 Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sun, 14 Jul 2019 22:35:19 -0700 Received: from NAM01-BN3-obe.outbound.protection.outlook.com (104.47.33.52) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Sun, 14 Jul 2019 22:35:19 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OUrsGjvEXv5ATMaLLALBGkubZvEpdNO1jUe4m6HEOExS3tGrMUeIU5nobq9kkpFr/Wle7s1D+V4lXivwRoObUlgmYtwfpUNWpUotgtLzcirp5kPklh+IyX85zrgkJTTXVLKiwRS2RUTMWxMeKVWT7f/cgLOT2m+V6mc/HxwXEJGGpMM7wiDhBL9no+3rGGHwTouXP7opm0LiDFZVSDw/eL2MEOLJbA4HVLa+K3PyPquAUJHz6UPf08qrhmn6xFSgApT7reE8EZEqK+6oOeLQ6c2gehbuL7PvtDBGsYcG7qrRn8dBN32DzAP6BOi9yM3R+OK0NJCBm9Mo92kZPzXsAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=; b=a6d4mYQgCGz8RhThAgQO+pad3+IwsBlJivgAj3DN0IzV0Qbcmptmd1MNVuD4oxx3AR5haUlBED0JPaiPiAfm6NWyWtvcmabF0nlZ6htEbC/rWnKyujha9DKlLaCN1+b0TXGIyH4kJjKQwjT/2WXyWeuOLLoTj4+j4Otv+UJOFZrJLeeYwfl6v98heKXAr0ghnEavkYMroKO1zP4/KzwJPGEPvjCuA0vIDxGj0HZPXFL9VuRnsybDoXeRE0jcM7sjaFsOyuG7qq2BE17HjWizinec59uOFmIahBjPLdBqFyuLWrsZZ2H8CmCc+6QlRBhX4PP/AGimfz8e7AVCD87nRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=marvell.com;dmarc=pass action=none header.from=marvell.com;dkim=pass header.d=marvell.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=; b=qbBVKcojzK4eZUBy8qpNjiNJTpf2EhzWqHAhZ4nOJo+uqyrda3t0v4MR3rHEXZWcLnncfqRnjTRE4TsyQJjsUkaHCfM44cMtLYULJTFhJSGWFIQ7iyU2hKDGbFC1o+vEnWwFjj0QCQpDkArUfhIgu2CESHlEVK3MHbx80pjPmbQ= Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by BYAPR18MB2453.namprd18.prod.outlook.com (20.179.92.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2073.14; Mon, 15 Jul 2019 05:35:14 +0000 Received: from BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862]) by BYAPR18MB2424.namprd18.prod.outlook.com ([fe80::2d42:12b6:aa2e:2862%4]) with mapi id 15.20.2073.012; Mon, 15 Jul 2019 05:35:14 +0000 From: Jerin Jacob Kollanukkaran To: Thomas Monjalon , "Hyong Youb Kim (hyonkim)" CC: David Marchand , "dev@dpdk.org" , "anatoly.burakov@intel.com" , "alex.williamson@redhat.com" , "maxime.coquelin@redhat.com" , "stephen@networkplumber.org" , "igor.russkikh@aquantia.com" , "pavel.belous@aquantia.com" , "allain.legacy@windriver.com" , "matt.peters@windriver.com" , "ravi1.kumar@amd.com" , Rasesh Mody , Shahed Shaikh , "ajit.khaparde@broadcom.com" , "somnath.kotur@broadcom.com" , "hemant.agrawal@nxp.com" , "shreyansh.jain@nxp.com" , "wenzhuo.lu@intel.com" , "mw@semihalf.com" , "mk@semihalf.com" , "gtzalik@amazon.com" , "evgenys@amazon.com" , "John Daley (johndale)" , "qi.z.zhang@intel.com" , "xiao.w.wang@intel.com" , "xuanziyang2@huawei.com" , "cloud.wangxiaoyun@huawei.com" , "zhouguoyang@huawei.com" , "beilei.xing@intel.com" , "jingjing.wu@intel.com" , "qiming.yang@intel.com" , "konstantin.ananyev@intel.com" , "alejandro.lucero@netronome.com" , "arybchenko@solarflare.com" , "tiwei.bie@intel.com" , "zhihong.wang@intel.com" , "yongwang@vmware.com" , "stable@dpdk.org" , Nithin Kumar Dabilpuram Thread-Topic: [dpdk-dev] [PATCH] vfio: fix interrupts race condition Thread-Index: AQHVNxvP0YZCfMFbb0SHaZ04OkExBqbEXHuAgAU6TYCAAGcUgIABMUrA Date: Mon, 15 Jul 2019 05:35:14 +0000 Message-ID: References: <1562071706-11009-1-git-send-email-david.marchand@redhat.com> <1796500.5oFe8j95cd@xps> <4647179.CTOrK8BQiK@xps> In-Reply-To: <4647179.CTOrK8BQiK@xps> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [14.140.231.66] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 79dbddd2-f02f-4eba-9ddd-08d708e63650 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:BYAPR18MB2453; x-ms-traffictypediagnostic: BYAPR18MB2453: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 00997889E7 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(396003)(366004)(39850400004)(346002)(376002)(189003)(199004)(478600001)(7696005)(68736007)(55016002)(66946007)(76116006)(66446008)(64756008)(66476007)(5660300002)(6436002)(66556008)(6506007)(6246003)(4326008)(7406005)(7416002)(107886003)(99286004)(8936002)(9686003)(316002)(2906002)(81156014)(81166006)(53936002)(229853002)(6116002)(3846002)(52536014)(25786009)(486006)(446003)(11346002)(7736002)(305945005)(476003)(102836004)(8676002)(45080400002)(186003)(256004)(86362001)(55236004)(33656002)(74316002)(76176011)(54906003)(66066001)(110136005)(14454004)(71200400001)(71190400001)(14444005)(26005); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB2453; H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Ay6QuXbdDGHWbr7/cuPTRKY1wHwKx5lhNwxDMsNRTt4CQdTAAWcM4VIeQJLpGyYlqT3b6hIdQWng7LIZgmChiSIXk7dzUqOlgj4p/q+PjCzQhYLhHtxnCcQp+8GV90OBrBC9z1wUc94jBqGfx7ZS0t4RXYXKY02q5P0uc4a4xxM4iwZYkBWVLFYxPZ2mjbkFmC6lhMcNV2DCCoI9omiGzU024ChsPzS+5qEOUlIxIgqYYgq2zl6F3juUtI9wzPq34iYIc4D+4W1CCEVxL5c2Ws8iZPkgq82d/UJhLkDQAaviZuc7PzHmMqZSoRfBkZHh7ObUG1x7WPiHm8rPiuc27H9QsQg6mUS6OPq8F4lfFGA9MVMSR1V0R8eOL5I7uPnYA5zL5muvhlPzN+DiQd36k4QurKSMsuNWkCMpgA/6Y4s= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 79dbddd2-f02f-4eba-9ddd-08d708e63650 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Jul 2019 05:35:14.0229 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jerinj@marvell.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2453 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8 definitions=2019-07-15_01:2019-07-12,2019-07-15 signatures=0 Subject: Re: [dpdk-dev] [PATCH] vfio: fix interrupts race condition X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > > > > > > This is a real bug which should be fixed in this release. > > > As the patch is quite big and needs a strong validation, I prefer > > > merging it quickly to give a lot of time before releasing 19.08-rc2. > > > The maintainers of all concerned PMDs are Cc. > > > Please make sure the interrupts are still working well with VFIO. > > > > > > Applied, thanks > > > > > > > [Apologies in advance if email format gets messed up. Forced to use > > outlook for the first time..] > > > > Hi, > > > > This commit breaks MSI-X + rxq interrupts. I think others are seeing > > the same error? > > > > sudo ~/dpdk/examples/l3fwd-power/build/l3fwd-power \ -c 0x1e -n 4 -w > > 0000:1a:00.0 --log-level=3Dpmd,debug -- -p 0x1 -P --config > "(0,0,2),(0,1,3),(0,2,4)" > > [...] > > EAL: Error enabling MSI-X interrupts for fd 35 > > > > A rough sequence of events goes like this. The above test is using 3 > > rxqs (3 interrupts). > > > > 1. During probe, pci_vfio_setup_interrupts() runs. > > This now does ioctl(VFIO_DEVICE_SET_IRQS) for the 1st efd > > (intr_handle->fd). > > > > ioctl does: > > - pci_enable_msix(1 vector) because this is the first time enabling > > interrupts. > > - request_irq(vector 0) > > > > 2. App configs > > The app sets port_conf.intr_conf.rxq=3D1, configs 3 rxqs, etc. > > > > 3. rte_eth_dev_start() > > PMD calls: > > - rte_intr_efd_enable() > > This creates 3 efds (intr_handle->nb_efd =3D 3). > > - rte_intr_enable() =3D> vfio_enable_msix() > > This does ioctl(VFIO_DEVICE_SET_IRQS) for the 3 efds. > > > > ioctl now needs to request_irq() for vectors 1, 2, 3 for the 3 new > > efds. It does not do another pci_enable_msix() as it has been done > > earlier. Before calling request_irq(), it sees that only 1 vector was > > enabled in earlier pci_enable_msix(), so it fails with EINVAL. > > > > We would need pci_enable_msix(4 vectors) for this to work > > (intr_handle->fd + 3 efds). > > > > Prior to this patch, VFIO_DEVICE_SET_IRQS is done only in > > vfio_enable_msix(). So, ioctl ends up doing pci_enable_msix(4 vectors) > > and request_irq() for each of the 4 efds, which completes > > successfully. > > > > Not an expert in this area.. Perhaps, defer enabling 1st efd > > (intr_handle->fd) until the first invocation of vfio_enable_msix(), so > > it knows the app wants to use 4 vectors in total? > > > > Also, vfio_disable_msix() looks a bit wrong. > > > > irq_set.flags =3D VFIO_IRQ_SET_DATA_NONE | > VFIO_IRQ_SET_ACTION_TRIGGER; > > irq_set.index =3D VFIO_PCI_MSIX_IRQ_INDEX; > > irq_set.start =3D RTE_INTR_VEC_RXTX_OFFSET; > > irq_set.count =3D intr_handle->nb_efd; > > > > This tells vfio-pci to simulate interrupts by triggering efds? To > > free_irq() specific efds, I think we need DATA_EVENTFD and set fd =3D > > -1. > > > > flags =3D DATA_EVENTFD | ACTION_TRIGGER > > data =3D [fd(-1), fd(-1), ...] > > > > I have not tested this part myself yet. We do see the following failure[1] on octeontx2 PMD with this patch. We will try to find a fix. irq_set =3D (struct vfio_irq_set *)irq_set_buf; irq_set->argsz =3D len; irq_set->start =3D 0; irq_set->count =3D intr_handle->max_intr; irq_set->flags =3D VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER; irq_set->index =3D VFIO_PCI_MSIX_IRQ_INDEX; fd_ptr =3D (int32_t *)&irq_set->data[0]; for (i =3D 0; i < irq_set->count; i++) fd_ptr[i] =3D -1; rc =3D ioctl(intr_handle->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_se= t); if (rc) otx2_err("Failed to set irqs vector rc=3D%d", rc); ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[1] =09 =09 =09 >=20 > Thanks for your detailed report Hyong. > Would you be able to propose a fix? >=20