From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8DF4AA04DB; Mon, 30 Nov 2020 18:20:53 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E5787C952; Mon, 30 Nov 2020 18:20:51 +0100 (CET) Received: from FRA01-MR2-obe.outbound.protection.outlook.com (mail-eopbgr90097.outbound.protection.outlook.com [40.107.9.97]) by dpdk.org (Postfix) with ESMTP id CD720C922 for ; Mon, 30 Nov 2020 18:20:50 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d2e3leSuj2F3/bqA/UBQKT+3gSwlfpd7pyoRoFXe30HnbIXJD48cycxxTllh5qODk95ZxCz+3XU4GeUnnJiIh4VX6wxu0qDzGeKuz8D8Q4GhBjdu5T2i4xxzRPrmVvgG3ExPs39hrUoZiO1R8n+/N10U2boR2bJ1MjsT3+5BA3H8O8C6TXr1MekNMPOaNlea8qaxxs+oMOZwVCQTqbWe+HDKtxejLUGyvAGUk/jgLZl9WL3loCjCr6waYeuOLWowIiQM5Q/kPemCuacbfsxZXgrfvcRbowpONGGA6v6QYqF/Lfm28P3+vF5YtqxsgVqgnCpuoNl9R1mRs6T/OalBzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VYsCuBpLluc/Pb2fC3f6UJSaWp8BEc0WlAFh3FODLeY=; b=lD8eDzVMNbwJn+XeqRiXItqjrXMGzN4OggXie2U/yoGVNzCAcq0exViHSx0qx57LVba71MVkC6XW5RaoUREzWY/AjHeHm5mlG6YC98qv+/YshyC9+lhw9Y+rJ3GKmJ/zMgh6hISX9kGLKs1PlkT8zWGQGrUT3NkJuclVq+w21H60OjCSt35Rr/LBPNV7HDE7jGQLroj8gmay8AbkynwRWCNBbSwatcMKRJF1naWrseaoHlZLhlV/jxoe+1NDs5U3QLYbdHn/D2zaC3c+D32FJKiuQhMl+TDYxw3fx8Acd/rpl9rtTmtyMGX3E+TNRDwjEtsJeYIqzHvKhiGcOQ8nng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ekinops.com; dmarc=pass action=none header.from=ekinops.com; dkim=pass header.d=ekinops.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ekinops.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VYsCuBpLluc/Pb2fC3f6UJSaWp8BEc0WlAFh3FODLeY=; b=au+o/VCy6yCCNiy/L6xfvG6mfffHsx9K6+DSwAmnH0++KfTZyBexFXpy+MBuTsb9q9HLP0sgIZ+QsWm/7pPTXaER3kC72Hw2O/WMtNZeCjjKEHld9aWCgVVvNl0YgLtZmGQ19hBMYkyzaH/33jcQsr0DtARiGTrOFtvzoZJ23+M= Received: from MRXP264MB0120.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:1b::10) by MRXP264MB0567.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:16::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.25; Mon, 30 Nov 2020 17:20:48 +0000 Received: from MRXP264MB0120.FRAP264.PROD.OUTLOOK.COM ([fe80::25e3:f7fb:eac4:f697]) by MRXP264MB0120.FRAP264.PROD.OUTLOOK.COM ([fe80::25e3:f7fb:eac4:f697%7]) with mapi id 15.20.3564.036; Mon, 30 Nov 2020 17:20:48 +0000 From: Renata Saiakhova To: Harman Kalra CC: Anatoly Burakov , Bruce Richardson , Ray Kinsella , Neil Horman , "dev@dpdk.org" Thread-Topic: [EXT] [PATCH v2 1/1] librte_eal: rte_intr_callback_unregister_sync() - wrapper around rte_intr_callback_unregister(). Thread-Index: AQHWdJ/8pxKe/WdWdUqRXrlPSb8KH6mt6suAgDOYJ9g= Date: Mon, 30 Nov 2020 17:20:48 +0000 Message-ID: References: <20200817140828.9769-1-Renata.Saiakhova@ekinops.com> <20200817140828.9769-2-Renata.Saiakhova@ekinops.com>, <20201028203632.GA137989@outlook.office365.com> In-Reply-To: <20201028203632.GA137989@outlook.office365.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=ekinops.com; x-originating-ip: [2a02:1811:c1f:7f00:ae72:c784:60ad:9d07] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d4b97ed6-6848-48ad-fce3-08d89554479c x-ms-traffictypediagnostic: MRXP264MB0567: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ZVKaJM/Yv0eH7WUzfadRVpVbGMohL51B2QLjyIA2EQnRH2kvWmW0h+qX11ZtQ3Sqkvm53xPHbrd54TiLCkNuZBBSjNKiEGJ4g/efWi2QzOmSXTPVfuunU0TYyLqlzXQvL97bRg8v+QtjZQdr3jI6bmbkldsWL/hfWzMIMoBkF6mS6OSgNOI3Vz0CufAp9NYlznrA8Lx30RjW+jft5Z65vwS/y2s2/LIitJ2bSAXWbBgomBw6gFMEn6iQWaXbbO9MJFqg7qh0bmXGgkS/hj32V2LUSe2AMCaSz+0Oi3DuWA44/4JS14bO71x2dq1qRQa6PdCixckH+epr35u9WeQsdg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MRXP264MB0120.FRAP264.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(4636009)(366004)(136003)(39850400004)(396003)(376002)(346002)(2906002)(19627405001)(54906003)(8676002)(7696005)(316002)(4326008)(6916009)(66476007)(52536014)(478600001)(86362001)(91956017)(5660300002)(83380400001)(8936002)(6506007)(76116006)(33656002)(66946007)(186003)(53546011)(55016002)(9686003)(66556008)(66446008)(64756008)(71200400001)(44832011); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: =?iso-8859-1?Q?LYlHS+pTiTGvZVKEZ3aIxDQxkiG10JkXEuk5vyh8pSObbErTVZp9ChzZXu?= =?iso-8859-1?Q?TFjONIPYNyQn6dalFyw3L+zExmnSKq9CAwO0IJhdDimKh2JkWWkzSc324a?= =?iso-8859-1?Q?pQpQ/4wFJxCmbl7ezhfgjYURAk98JdQczK2TE1OIsZIA2bXc+rfxQ5lYrR?= =?iso-8859-1?Q?f6FZlhsbicgxSNO2uxzMS9oCuuUbnwaY0/Oi96ds9ZpsMiR+wefm0yrRAx?= =?iso-8859-1?Q?hs7MAAS5KRDQIY3/Cg1g6AVPFtJqPHO603i5LqnZE/zaPcf0SKGgZI28nr?= =?iso-8859-1?Q?uUY+Pj56CANDQf7qn0TGFburFRAtz2oJuAzG068J534usnqSQOsAHREBLQ?= =?iso-8859-1?Q?W1rWJqc6uE2S1KlQHHy3BD9PoIKXon7DspMYqgg3UAnbQHJg8E4GCnvyWM?= =?iso-8859-1?Q?6EG9qgFqRFja2BgpuTmoKLcqRI7IOG/WhqCQUbPR/HRmELKDl7x6sf3eTD?= =?iso-8859-1?Q?jG5inovyzHByed18HgAK768OHcb2oG22fLDbdoddJo/SHUQEjw1ASX6Ctr?= =?iso-8859-1?Q?IFlV4N7segj1oJ35K+0B2ThZy18rjNZleeqkvia6XTqBXPkmha9Z9CQeEA?= =?iso-8859-1?Q?Ro3nLyZeJh3vRrvlMdxq8yYTLjNKnzY9QRNf/CXF3Lb4sc8SNuyo54nhQh?= =?iso-8859-1?Q?NsWflOOeP7NI+TVY5U6gZ9vYwPWH2RAJKGEmBu0wmCD1Y7YMi/FRhRcn+y?= =?iso-8859-1?Q?w0j2dgvn2b8xgweB4zA6qM5hHJA8tTTb1taX++nxe2YEVoHw6FkxPxSZOE?= =?iso-8859-1?Q?so3wnEyLPB29q+Oi/ySNFWNepHRMVtk6p05YEg2dWcXzHhZrxO734WhNQS?= =?iso-8859-1?Q?1m7TCl/oQ1+9G7mxJRIi8ce2L3wRbBQSE1YyL8VxpW3dA6/fOJSW4nkMYI?= =?iso-8859-1?Q?vlrMcW4Nn+6Ikm/mjtp81enn8D/QF1pgW0moinwgMeVmgBFMd1oAwf5tQb?= =?iso-8859-1?Q?F40ypdAhEx3crNJBmjWkEp6G+2/jchkS53C+lZ56EN+Nd4tJKAf5uQQazL?= =?iso-8859-1?Q?cpFP3BdgApX4KIfaxpPRuoQ2C3ggeuyRDd6g/MSuerrOZhRkH/WbFwj9GX?= =?iso-8859-1?Q?2nMozTnw8k+kYEQBgDCHmq4=3D?= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: ekinops.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MRXP264MB0120.FRAP264.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: d4b97ed6-6848-48ad-fce3-08d89554479c X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Nov 2020 17:20:48.3387 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: f57b78a6-c654-4771-a72f-837275f46179 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: k/2MOHQY6qTIzzUmd2GAc/8P46Pn1iq3wWufBConSG+3RFJwp4EqGtfANtWHv4MCn/ExWEgDzfsM1Frf33pSzNgoZNsRYhDmKn4Z3VEaVL0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRXP264MB0567 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [EXT] [PATCH v2 1/1] librte_eal: rte_intr_callback_unregister_sync() - wrapper around rte_intr_callback_unregister(). X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Harman, sorry for late reply... Yes, indeed, this is a race between an application which calls rte_dev_remo= ve() and a kernel event which is sent as a result of unbinding the device f= rom vfio_pci driver. (dpdk-devbind.py -u 0000:05:00.0) rte_intr_callback_unregister() may fail and return -EAGAIN, if an interrupt= source (kernel) has some active callbacks at the moment. As a result, the = callback (req notifier) can never be unregistered, and vfio_req_intr_handle.fd can never be closed. The kernel will continuously try to notify the user space using req notifie= r, but as the device is already removed, in this case it even cannot find a= bus for that device, below is the log which illustrates it: EAL: fail to unregister req notifier handler. EAL: fail to disable req notifier. dpdk_disconnect 1545: Device '0000:05:00.0' has been removed and detached dpdk_disconnect 1557: All devices shared with device '0000:05:00.0' have be= en detached EAL: Cannot find bus for device (05:00.0) EAL: Cannot find bus for device (05:00.0) EAL: Cannot find bus for device (05:00.0) EAL: Cannot find bus for device (05:00.0) EAL: Cannot find bus for device (05:00.0) etc. This continues eternally, and application stops to work properly. So, at least the retry logic should be put somewhere to avoid this kind of = race. Or bus->hot_unplug_handler(dev) called from pci_vfio_req_handler() sh= ould do some work to release the above resources. Regarding the tight polling loop in the patch and fixed retry logic to avo= id infinite looping, what could be an option? As it continues to loop only = in -EAGAIN case, which means kernel event is processed, doesn't it guarante= e that it won't last forever? Kind regards, Renata ________________________________ From: Harman Kalra Sent: Wednesday, October 28, 2020 9:36 PM To: Renata Saiakhova Cc: Anatoly Burakov ; Bruce Richardson ; Ray Kinsella ; Neil Horman ; dev@dpdk.org Subject: Re: [EXT] [PATCH v2 1/1] librte_eal: rte_intr_callback_unregister_= sync() - wrapper around rte_intr_callback_unregister(). On Mon, Aug 17, 2020 at 04:08:27PM +0200, Renata Saiakhova wrote: > External Email > > ---------------------------------------------------------------------- > Avoid race with unregister interrupt hanlder if interrupt > source has some active callbacks at the moment, use wrapper > around rte_intr_callback_unregister() to check for -EAGAIN > return value and to loop until rte_intr_callback_unregister() > succeeds. > Hi Renata, Just trying to understand the scenario, as you mentioned "while removing the device by rte_dev_remove()" are you calling rte_eal_hotplug_remove or kernel has sent an event to remove the device. As far as I know vfio notifier mechanism is used by kernel vfio driver to notify user to release the resources and as you are observing EAGAIN means same callback is executing. Regarding the tight polling loop in the patch, I think its good to have a fixed retry logic to avoid any unidentified corner case which might lead to infinite looping. Thanks Harman > Signed-off-by: Renata Saiakhova > --- > drivers/bus/pci/linux/pci_vfio.c | 2 +- > lib/librte_eal/freebsd/eal_interrupts.c | 12 ++++++++++++ > lib/librte_eal/include/rte_interrupts.h | 25 +++++++++++++++++++++++++ > lib/librte_eal/linux/eal_interrupts.c | 12 ++++++++++++ > lib/librte_eal/rte_eal_version.map | 1 + > 5 files changed, 51 insertions(+), 1 deletion(-) > > diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci= _vfio.c > index 07e072e13..a4bfdf553 100644 > --- a/drivers/bus/pci/linux/pci_vfio.c > +++ b/drivers/bus/pci/linux/pci_vfio.c > @@ -415,7 +415,7 @@ pci_vfio_disable_notifier(struct rte_pci_device *dev) > return -1; > } > > - ret =3D rte_intr_callback_unregister(&dev->vfio_req_intr_handle, > + ret =3D rte_intr_callback_unregister_sync(&dev->vfio_req_intr_handl= e, > pci_vfio_req_handler, > (void *)&dev->device); > if (ret < 0) { > diff --git a/lib/librte_eal/freebsd/eal_interrupts.c b/lib/librte_eal/fre= ebsd/eal_interrupts.c > index 6d53d33c8..7d99bdaff 100644 > --- a/lib/librte_eal/freebsd/eal_interrupts.c > +++ b/lib/librte_eal/freebsd/eal_interrupts.c > @@ -345,6 +345,18 @@ rte_intr_callback_unregister(const struct rte_intr_h= andle *intr_handle, > return ret; > } > > +int > +rte_intr_callback_unregister_sync(const struct rte_intr_handle *intr_han= dle, > + rte_intr_callback_fn cb_fn, void *cb_arg) > +{ > + int ret =3D 0; > + > + while ((ret =3D rte_intr_callback_unregister(intr_handle, cb_fn, cb= _arg)) =3D=3D -EAGAIN) > + rte_pause(); > + > + return ret; > +} > + > int > rte_intr_enable(const struct rte_intr_handle *intr_handle) > { > diff --git a/lib/librte_eal/include/rte_interrupts.h b/lib/librte_eal/inc= lude/rte_interrupts.h > index e3b406abc..cc3bf45d8 100644 > --- a/lib/librte_eal/include/rte_interrupts.h > +++ b/lib/librte_eal/include/rte_interrupts.h > @@ -94,6 +94,31 @@ rte_intr_callback_unregister_pending(const struct rte_= intr_handle *intr_handle, > rte_intr_callback_fn cb_fn, void *cb_arg, > rte_intr_unregister_callback_fn ucb_fn); > > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change without prior notice > + * > + * Loop until rte_intr_callback_unregister() succeeds. > + * After a call to this function, > + * the callback provided by the specified interrupt handle is unregister= ed. > + * > + * @param intr_handle > + * pointer to the interrupt handle. > + * @param cb > + * callback address. > + * @param cb_arg > + * address of parameter for callback, (void *)-1 means to remove all > + * registered which has the same callback address. > + * > + * @return > + * - On success, return the number of callback entities removed. > + * - On failure, a negative value. > + */ > +__rte_experimental > +int > +rte_intr_callback_unregister_sync(const struct rte_intr_handle *intr_han= dle, > + rte_intr_callback_fn cb, void *cb_arg); > + > /** > * It enables the interrupt for the specified handle. > * > diff --git a/lib/librte_eal/linux/eal_interrupts.c b/lib/librte_eal/linux= /eal_interrupts.c > index 13db5c4e8..c99d5dbd4 100644 > --- a/lib/librte_eal/linux/eal_interrupts.c > +++ b/lib/librte_eal/linux/eal_interrupts.c > @@ -662,6 +662,18 @@ rte_intr_callback_unregister(const struct rte_intr_h= andle *intr_handle, > return ret; > } > > +int > +rte_intr_callback_unregister_sync(const struct rte_intr_handle *intr_han= dle, > + rte_intr_callback_fn cb_fn, void *cb_arg) > +{ > + int ret =3D 0; > + > + while ((ret =3D rte_intr_callback_unregister(intr_handle, cb_fn, cb= _arg)) =3D=3D -EAGAIN) > + rte_pause(); > + > + return ret; > +} > + > int > rte_intr_enable(const struct rte_intr_handle *intr_handle) > { > diff --git a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_= version.map > index bf0c17c23..b1d824f59 100644 > --- a/lib/librte_eal/rte_eal_version.map > +++ b/lib/librte_eal/rte_eal_version.map > @@ -325,6 +325,7 @@ EXPERIMENTAL { > rte_fbarray_find_rev_biggest_free; > rte_fbarray_find_rev_biggest_used; > rte_intr_callback_unregister_pending; > + rte_intr_callback_unregister_sync; > rte_realloc_socket; > > # added in 19.08 > -- > 2.17.2 >