From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by inbox.dpdk.org (Postfix) with ESMTP id 32C21A0471
for <public@inbox.dpdk.org>; Mon, 15 Jul 2019 07:35:31 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 44935325F;
Mon, 15 Jul 2019 07:35:30 +0200 (CEST)
Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com
[67.231.148.174]) by dpdk.org (Postfix) with ESMTP id B85E13256;
Mon, 15 Jul 2019 07:35:27 +0200 (CEST)
Received: from pps.filterd (m0045849.ppops.net [127.0.0.1])
by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id
x6F5ZK4m014239; Sun, 14 Jul 2019 22:35:20 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com;
h=from : to : cc :
subject : date : message-id : references : in-reply-to : content-type :
content-transfer-encoding : mime-version; s=pfpt0818;
bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=;
b=Pia1qytPnO6EGjKo4iurnx1U89myiOuHgpzXgMjHe0JXh+RntbE8y5dQAuFJXib0+RJW
NRvoAwxn9S/m6v7P0ozcLbCKHoIAdwVx29Bzjfjs2D7fNID+kluZ499b9CCgm0m+BLn5
kVu4JqjA5cfuPbwP9NO+W9DZLCfjBytm8a89Td6mo20DfxEQrNKZA/da5sQ5NAIW8w0k
8PpsyuMphQlO/kQYHwojV4cuygw4GOZ8DgdT71nY4y/YFifqoXkcKAgFubgQ/SGPTDt4
UldbMHJsTDigX+3VhYlikAA6vbL31Vnj58bCHIISF4nB/O2D1yeyq8hT/NA24sMqX4cP RA==
Received: from sc-exch03.marvell.com ([199.233.58.183])
by mx0a-0016f401.pphosted.com with ESMTP id 2tqcnpp71k-2
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT);
Sun, 14 Jul 2019 22:35:20 -0700
Received: from SC-EXCH04.marvell.com (10.93.176.84) by SC-EXCH03.marvell.com
(10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sun, 14 Jul
2019 22:35:19 -0700
Received: from NAM01-BN3-obe.outbound.protection.outlook.com (104.47.33.52) by
SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server
(TLS) id
15.0.1367.3 via Frontend Transport; Sun, 14 Jul 2019 22:35:19 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
b=OUrsGjvEXv5ATMaLLALBGkubZvEpdNO1jUe4m6HEOExS3tGrMUeIU5nobq9kkpFr/Wle7s1D+V4lXivwRoObUlgmYtwfpUNWpUotgtLzcirp5kPklh+IyX85zrgkJTTXVLKiwRS2RUTMWxMeKVWT7f/cgLOT2m+V6mc/HxwXEJGGpMM7wiDhBL9no+3rGGHwTouXP7opm0LiDFZVSDw/eL2MEOLJbA4HVLa+K3PyPquAUJHz6UPf08qrhmn6xFSgApT7reE8EZEqK+6oOeLQ6c2gehbuL7PvtDBGsYcG7qrRn8dBN32DzAP6BOi9yM3R+OK0NJCBm9Mo92kZPzXsAw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
s=arcselector9901;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=;
b=a6d4mYQgCGz8RhThAgQO+pad3+IwsBlJivgAj3DN0IzV0Qbcmptmd1MNVuD4oxx3AR5haUlBED0JPaiPiAfm6NWyWtvcmabF0nlZ6htEbC/rWnKyujha9DKlLaCN1+b0TXGIyH4kJjKQwjT/2WXyWeuOLLoTj4+j4Otv+UJOFZrJLeeYwfl6v98heKXAr0ghnEavkYMroKO1zP4/KzwJPGEPvjCuA0vIDxGj0HZPXFL9VuRnsybDoXeRE0jcM7sjaFsOyuG7qq2BE17HjWizinec59uOFmIahBjPLdBqFyuLWrsZZ2H8CmCc+6QlRBhX4PP/AGimfz8e7AVCD87nRQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass
smtp.mailfrom=marvell.com;dmarc=pass action=none
header.from=marvell.com;dkim=pass header.d=marvell.com;arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=marvell.onmicrosoft.com; s=selector2-marvell-onmicrosoft-com;
h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
bh=MAZtxIvZ1wv+sFVNHCefAcDRFJa9sgeHwLIvo7NAtJE=;
b=qbBVKcojzK4eZUBy8qpNjiNJTpf2EhzWqHAhZ4nOJo+uqyrda3t0v4MR3rHEXZWcLnncfqRnjTRE4TsyQJjsUkaHCfM44cMtLYULJTFhJSGWFIQ7iyU2hKDGbFC1o+vEnWwFjj0QCQpDkArUfhIgu2CESHlEVK3MHbx80pjPmbQ=
Received: from BYAPR18MB2424.namprd18.prod.outlook.com (20.179.91.149) by
BYAPR18MB2453.namprd18.prod.outlook.com (20.179.92.22) with Microsoft SMTP
Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
15.20.2073.14; Mon, 15 Jul 2019 05:35:14 +0000
Received: from BYAPR18MB2424.namprd18.prod.outlook.com
([fe80::2d42:12b6:aa2e:2862]) by BYAPR18MB2424.namprd18.prod.outlook.com
([fe80::2d42:12b6:aa2e:2862%4]) with mapi id 15.20.2073.012; Mon, 15 Jul 2019
05:35:14 +0000
From: Jerin Jacob Kollanukkaran <jerinj@marvell.com>
To: Thomas Monjalon <thomas@monjalon.net>, "Hyong Youb Kim (hyonkim)"
<hyonkim@cisco.com>
CC: David Marchand <david.marchand@redhat.com>, "dev@dpdk.org" <dev@dpdk.org>,
"anatoly.burakov@intel.com" <anatoly.burakov@intel.com>,
"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
"stephen@networkplumber.org" <stephen@networkplumber.org>,
"igor.russkikh@aquantia.com" <igor.russkikh@aquantia.com>,
"pavel.belous@aquantia.com" <pavel.belous@aquantia.com>,
"allain.legacy@windriver.com" <allain.legacy@windriver.com>,
"matt.peters@windriver.com" <matt.peters@windriver.com>,
"ravi1.kumar@amd.com" <ravi1.kumar@amd.com>,
Rasesh Mody <rmody@marvell.com>, Shahed Shaikh <shshaikh@marvell.com>,
"ajit.khaparde@broadcom.com" <ajit.khaparde@broadcom.com>,
"somnath.kotur@broadcom.com" <somnath.kotur@broadcom.com>,
"hemant.agrawal@nxp.com" <hemant.agrawal@nxp.com>,
"shreyansh.jain@nxp.com" <shreyansh.jain@nxp.com>,
"wenzhuo.lu@intel.com" <wenzhuo.lu@intel.com>, "mw@semihalf.com"
<mw@semihalf.com>, "mk@semihalf.com" <mk@semihalf.com>,
"gtzalik@amazon.com" <gtzalik@amazon.com>,
"evgenys@amazon.com" <evgenys@amazon.com>, "John Daley
(johndale)" <johndale@cisco.com>, "qi.z.zhang@intel.com"
<qi.z.zhang@intel.com>, "xiao.w.wang@intel.com" <xiao.w.wang@intel.com>,
"xuanziyang2@huawei.com" <xuanziyang2@huawei.com>,
"cloud.wangxiaoyun@huawei.com" <cloud.wangxiaoyun@huawei.com>,
"zhouguoyang@huawei.com" <zhouguoyang@huawei.com>, "beilei.xing@intel.com"
<beilei.xing@intel.com>, "jingjing.wu@intel.com" <jingjing.wu@intel.com>,
"qiming.yang@intel.com" <qiming.yang@intel.com>,
"konstantin.ananyev@intel.com" <konstantin.ananyev@intel.com>,
"alejandro.lucero@netronome.com" <alejandro.lucero@netronome.com>,
"arybchenko@solarflare.com" <arybchenko@solarflare.com>,
"tiwei.bie@intel.com" <tiwei.bie@intel.com>, "zhihong.wang@intel.com"
<zhihong.wang@intel.com>, "yongwang@vmware.com" <yongwang@vmware.com>,
"stable@dpdk.org" <stable@dpdk.org>, Nithin Kumar Dabilpuram
<ndabilpuram@marvell.com>
Thread-Topic: [dpdk-dev] [PATCH] vfio: fix interrupts race condition
Thread-Index: AQHVNxvP0YZCfMFbb0SHaZ04OkExBqbEXHuAgAU6TYCAAGcUgIABMUrA
Date: Mon, 15 Jul 2019 05:35:14 +0000
Message-ID: <BYAPR18MB2424568FDD38F18548CC5B9CC8CF0@BYAPR18MB2424.namprd18.prod.outlook.com>
References: <1562071706-11009-1-git-send-email-david.marchand@redhat.com>
<1796500.5oFe8j95cd@xps>
<MWHPR11MB18392141DEEF5C77A761FAD0BFCC0@MWHPR11MB1839.namprd11.prod.outlook.com>
<4647179.CTOrK8BQiK@xps>
In-Reply-To: <4647179.CTOrK8BQiK@xps>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [14.140.231.66]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 79dbddd2-f02f-4eba-9ddd-08d708e63650
x-microsoft-antispam: BCL:0; PCL:0;
RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020);
SRVR:BYAPR18MB2453;
x-ms-traffictypediagnostic: BYAPR18MB2453:
x-microsoft-antispam-prvs: <BYAPR18MB24532F58BB34FE9623884C93C8CF0@BYAPR18MB2453.namprd18.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 00997889E7
x-forefront-antispam-report: SFV:NSPM;
SFS:(10009020)(4636009)(136003)(396003)(366004)(39850400004)(346002)(376002)(189003)(199004)(478600001)(7696005)(68736007)(55016002)(66946007)(76116006)(66446008)(64756008)(66476007)(5660300002)(6436002)(66556008)(6506007)(6246003)(4326008)(7406005)(7416002)(107886003)(99286004)(8936002)(9686003)(316002)(2906002)(81156014)(81166006)(53936002)(229853002)(6116002)(3846002)(52536014)(25786009)(486006)(446003)(11346002)(7736002)(305945005)(476003)(102836004)(8676002)(45080400002)(186003)(256004)(86362001)(55236004)(33656002)(74316002)(76176011)(54906003)(66066001)(110136005)(14454004)(71200400001)(71190400001)(14444005)(26005);
DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR18MB2453;
H:BYAPR18MB2424.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en;
PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: marvell.com does not designate
permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: Ay6QuXbdDGHWbr7/cuPTRKY1wHwKx5lhNwxDMsNRTt4CQdTAAWcM4VIeQJLpGyYlqT3b6hIdQWng7LIZgmChiSIXk7dzUqOlgj4p/q+PjCzQhYLhHtxnCcQp+8GV90OBrBC9z1wUc94jBqGfx7ZS0t4RXYXKY02q5P0uc4a4xxM4iwZYkBWVLFYxPZ2mjbkFmC6lhMcNV2DCCoI9omiGzU024ChsPzS+5qEOUlIxIgqYYgq2zl6F3juUtI9wzPq34iYIc4D+4W1CCEVxL5c2Ws8iZPkgq82d/UJhLkDQAaviZuc7PzHmMqZSoRfBkZHh7ObUG1x7WPiHm8rPiuc27H9QsQg6mUS6OPq8F4lfFGA9MVMSR1V0R8eOL5I7uPnYA5zL5muvhlPzN+DiQd36k4QurKSMsuNWkCMpgA/6Y4s=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 79dbddd2-f02f-4eba-9ddd-08d708e63650
X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Jul 2019 05:35:14.0229 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: jerinj@marvell.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR18MB2453
X-OriginatorOrg: marvell.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:5.22.84,1.0.8
definitions=2019-07-15_01:2019-07-12,2019-07-15 signatures=0
Subject: Re: [dpdk-dev] [PATCH] vfio: fix interrupts race condition
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
> > >
> > > This is a real bug which should be fixed in this release.
> > > As the patch is quite big and needs a strong validation, I prefer
> > > merging it quickly to give a lot of time before releasing 19.08-rc2.
> > > The maintainers of all concerned PMDs are Cc.
> > > Please make sure the interrupts are still working well with VFIO.
> > >
> > > Applied, thanks
> > >
> >
> > [Apologies in advance if email format gets messed up. Forced to use
> > outlook for the first time..]
> >
> > Hi,
> >
> > This commit breaks MSI-X + rxq interrupts. I think others are seeing
> > the same error?
> >
> > sudo ~/dpdk/examples/l3fwd-power/build/l3fwd-power \ -c 0x1e -n 4 -w
> > 0000:1a:00.0 --log-level=3Dpmd,debug -- -p 0x1 -P --config
> "(0,0,2),(0,1,3),(0,2,4)"
> > [...]
> > EAL: Error enabling MSI-X interrupts for fd 35
> >
> > A rough sequence of events goes like this. The above test is using 3
> > rxqs (3 interrupts).
> >
> > 1. During probe, pci_vfio_setup_interrupts() runs.
> > This now does ioctl(VFIO_DEVICE_SET_IRQS) for the 1st efd
> > (intr_handle->fd).
> >
> > ioctl does:
> > - pci_enable_msix(1 vector) because this is the first time enabling
> > interrupts.
> > - request_irq(vector 0)
> >
> > 2. App configs
> > The app sets port_conf.intr_conf.rxq=3D1, configs 3 rxqs, etc.
> >
> > 3. rte_eth_dev_start()
> > PMD calls:
> > - rte_intr_efd_enable()
> > This creates 3 efds (intr_handle->nb_efd =3D 3).
> > - rte_intr_enable() =3D> vfio_enable_msix()
> > This does ioctl(VFIO_DEVICE_SET_IRQS) for the 3 efds.
> >
> > ioctl now needs to request_irq() for vectors 1, 2, 3 for the 3 new
> > efds. It does not do another pci_enable_msix() as it has been done
> > earlier. Before calling request_irq(), it sees that only 1 vector was
> > enabled in earlier pci_enable_msix(), so it fails with EINVAL.
> >
> > We would need pci_enable_msix(4 vectors) for this to work
> > (intr_handle->fd + 3 efds).
> >
> > Prior to this patch, VFIO_DEVICE_SET_IRQS is done only in
> > vfio_enable_msix(). So, ioctl ends up doing pci_enable_msix(4 vectors)
> > and request_irq() for each of the 4 efds, which completes
> > successfully.
> >
> > Not an expert in this area.. Perhaps, defer enabling 1st efd
> > (intr_handle->fd) until the first invocation of vfio_enable_msix(), so
> > it knows the app wants to use 4 vectors in total?
> >
> > Also, vfio_disable_msix() looks a bit wrong.
> >
> > irq_set.flags =3D VFIO_IRQ_SET_DATA_NONE |
> VFIO_IRQ_SET_ACTION_TRIGGER;
> > irq_set.index =3D VFIO_PCI_MSIX_IRQ_INDEX;
> > irq_set.start =3D RTE_INTR_VEC_RXTX_OFFSET;
> > irq_set.count =3D intr_handle->nb_efd;
> >
> > This tells vfio-pci to simulate interrupts by triggering efds? To
> > free_irq() specific efds, I think we need DATA_EVENTFD and set fd =3D
> > -1.
> >
> > flags =3D DATA_EVENTFD | ACTION_TRIGGER
> > data =3D [fd(-1), fd(-1), ...]
> >
> > I have not tested this part myself yet.
We do see the following failure[1] on octeontx2 PMD with this patch.
We will try to find a fix.
irq_set =3D (struct vfio_irq_set *)irq_set_buf;
irq_set->argsz =3D len;
irq_set->start =3D 0;
irq_set->count =3D intr_handle->max_intr;
irq_set->flags =3D VFIO_IRQ_SET_DATA_EVENTFD |
VFIO_IRQ_SET_ACTION_TRIGGER;
irq_set->index =3D VFIO_PCI_MSIX_IRQ_INDEX;
fd_ptr =3D (int32_t *)&irq_set->data[0];
for (i =3D 0; i < irq_set->count; i++)
fd_ptr[i] =3D -1;
rc =3D ioctl(intr_handle->vfio_dev_fd, VFIO_DEVICE_SET_IRQS, irq_se=
t);
if (rc)
otx2_err("Failed to set irqs vector rc=3D%d", rc);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[1] =09
=09
=09
>=20
> Thanks for your detailed report Hyong.
> Would you be able to propose a fix?
>=20