From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0BD17A04FD; Mon, 23 May 2022 09:54:34 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E81E04014F; Mon, 23 May 2022 09:54:33 +0200 (CEST) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mails.dpdk.org (Postfix) with ESMTP id 4BC3A40041 for ; Mon, 23 May 2022 09:54:32 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1653292472; x=1684828472; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=gGG50+R/XrW6RfNmyFyFF0VEQYBe+zbsG30DckIf/L8=; b=d114Lep2dXa6A/sEjDZFq4Ybh+lrS3H8WNHE1Pcb2vuaWeVMIPdiUyt8 NycgqpK9eSE3BHcRiQNY+xYz7HPD9j842Q9dYIzwZcXEZ0C1Ny3mYoXQh 5938q7XYrpKNDVIw9MDSLthf0yvDDHuHY+amSiNOIN+wkVNUtcPN8q0EY yRI/UMchx+Y3BzSyKYnOviFWmHxZWcftRFlHTxVaneGOnHRPtfzg3VXXs qRcQQrx+Olbs7lUFxDSCBqTk+wrb4ZOlIfjtgRyjHSRWtjr7TNbiARsS7 YkoFpx07Bwj2+l5+gAD2pKehd1y81rcN8b0Ozbz3GtQL/r2T2yd8odm8d Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10355"; a="270710921" X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="270710921" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 May 2022 00:54:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,246,1647327600"; d="scan'208";a="547830138" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by orsmga006.jf.intel.com with ESMTP; 23 May 2022 00:54:22 -0700 Received: from fmsmsx605.amr.corp.intel.com (10.18.126.85) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27; Mon, 23 May 2022 00:54:22 -0700 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.27 via Frontend Transport; Mon, 23 May 2022 00:54:22 -0700 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (104.47.51.45) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.27; Mon, 23 May 2022 00:54:22 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SOx74X0CXpAFWFjKOxW+8+hRbcOCJ4N9F50+heWSybGUT/oRr9oQgkfREIClZkZ+jbc5vk96PoCeOGVqOgB42SPezvKQ6iAIU13vjnOrpia7ZW/nngi6KSjpPUwJ86pPNqrtpAu1CyDeYa/GS6JsAMl/WdmHjsIZXPNJvf+EMa6J3q4ld2YisTozCzwp3VhFFPxj5GT3VRRKXRMnJO8e28XQlxtB2lqp7YfV/lQa1gB8FeJDdzhYbHDteuEKBPOeyaPTDjBht70ekU3n1znF1PKTie39pldFbGYb9pzb2wSIbJt608iBwO33KUzy7g3QxGl023MgfV27ObHGRoTW5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aQwOVwZFQ8ZY/e5kvjoqbHV0Mw62sthwTnie55a20kQ=; b=oSrg9+9fxi58IloHPUsWaBcR2YOmyI6v7PNlqdHF9aElz3LVWMGU3UlXr0EE03PiP0ZFguR8wOc1uoE3/r8S7ZnNLERPYhKqQLy21SY+GWSA0u34xo80NqyvienxlcvZkksHN5GNIkbj4oDDfhO+p6dZfhnFc4vFpdn++Ai+WP2z3Vl8hnOg0j8ieqWleocpKUCTo3iJiir0gH1i/rZ7SCxmaRNoO56hpY7GCTLPMpBtHKiTykjtUdtArgqhn1NYUw1/h9AZ/Pq7fr+/DP5Vu+M4b8o6d56TxtG2fhynwbsXNCZB5zkCk/7Eg3XghW2jWFpy7kVDOe3XUv+ZXasv+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SN6PR11MB3504.namprd11.prod.outlook.com (2603:10b6:805:d0::17) by PH0PR11MB5596.namprd11.prod.outlook.com (2603:10b6:510:eb::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5273.13; Mon, 23 May 2022 07:54:20 +0000 Received: from SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f]) by SN6PR11MB3504.namprd11.prod.outlook.com ([fe80::54e1:e802:17c8:93f%7]) with mapi id 15.20.5273.023; Mon, 23 May 2022 07:54:20 +0000 From: "Xia, Chenbo" To: "Pei, Andy" , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "Cao, Gang" , "Liu, Changpeng" , "Xu, Rosen" , "Xiao, QimaiX" Subject: RE: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space Thread-Topic: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config space Thread-Index: AQHYardaXBGi2BX2FUmCeEYgApqK6K0sHhBA Date: Mon, 23 May 2022 07:54:20 +0000 Message-ID: References: <1643093258-47258-2-git-send-email-andy.pei@intel.com> <1652876035-70513-1-git-send-email-andy.pei@intel.com> <1652876035-70513-13-git-send-email-andy.pei@intel.com> In-Reply-To: <1652876035-70513-13-git-send-email-andy.pei@intel.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0b46f57f-d630-4b09-c140-08da3c9171c3 x-ms-traffictypediagnostic: PH0PR11MB5596:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: gPdCNY/R6H1drLqzwhUlsi614aZVcoHz8mCmfe+9fGmwo1CvHbQBfgrFvMSNqqYoYtZJwoaMosQFbzrrVJwN8LTjr6VA/lye0QI9YX0YLf1quNy6v9wp0+BM4gzr3DSvGG6j9VjnlcG+SSfXNghd5EVRaWotqFEyz6MV+MxewK9LFFb7n9JVzQQY8PZtZgxeYlhghQN9cfmVWXmA2kWuKzk83ZywRksoXK5xJuNy6jDoDgTklvaeGp/5Yma8s1uw++9NjjxNhKDfopdWZ7SV1bVGU9ls4jM6+jGsNLmfbZuPBhiJPXJvRUjjj77UZIbHRnnpC1aK9mQK8gHiyaQQGKsiNFHc1c9gPrb7gbkuMR5goMuVeWzxHqgzNgAHHcs4JXvHZYv4ngl97201VAuu37Q0x1q59jI9WRNaF50CpvbOiewjNIpMuz0irOZGxyBqCQ+PdnNVVjkIXgQ1LBAMo2Kn0pCdUJK+tQDYDDfjxM5kchk9RTKrUPrNsAz6K3wVz8iKrONkzK9/qE3+y1PZlNdBjE7Ong46mxu0v1ImGq242QdBBR2gzUotG1OvGvzLDLxwxPWgXdZzDUSXDzBVL86MSYkUZB+uxw6BvN8X1A9Db5LIhV39+ToMaO1SY+9eYAhe9sVNkSkD9mOMD0jx7T4U03gK5xaw5FcKva4prfHYD0rSp3PhR1wS/qjnL5p5vpdupTf3NSuaAxcurpqIxQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SN6PR11MB3504.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(83380400001)(26005)(122000001)(9686003)(66556008)(66476007)(76116006)(66946007)(110136005)(53546011)(82960400001)(33656002)(7696005)(316002)(64756008)(8676002)(4326008)(86362001)(508600001)(6506007)(54906003)(66446008)(2906002)(38100700002)(55016003)(5660300002)(107886003)(186003)(52536014)(38070700005)(8936002)(71200400001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?ZdvEq60LErx1C1PDHoZ1QGkqviu4JqU36Nn1hdPbi8plVUFuS8YibcyVGRWt?= =?us-ascii?Q?3OVFrY35M7arnb5iMy7/TXoEY/VZiJOm4h9CPRa5pd2ksZRJNgBcxN5vc5nR?= =?us-ascii?Q?tP4skhHJ/8J0oFQPHYFAMaE0vXxo5Yo+VfbnMn2rvr4S6jbCprB0jjzmOkX1?= =?us-ascii?Q?r73yr0O0V/+JaTXDzwfR9wSNfCLV1yi0+MwWH3Q1iFl/BzEkGrc4lCkGkL13?= =?us-ascii?Q?btk9KQFJWCcLtQvLJlYJP+gLWOSlf06mfL2duOJD7I4FzNtBzQPng3CQcK+K?= =?us-ascii?Q?/Z4zq8KCALLdNT6qb767RKXi6uGiypFCdaY9sKjRYDhB8tIRXhCrC0t57fOl?= =?us-ascii?Q?sG/nqFfqVm6m2ffm+6gBLoI614KKKTL0G2fJ+ZtHxTA65IgDLe3zZhVJEQAK?= =?us-ascii?Q?DBa4tV63iAgNmjvUSwLklCWc2QLad5o8cwYsxtgCSTS4MNUepBTTG01NMfv2?= =?us-ascii?Q?YyGUEZ/R0SC983chOwS2/UnYsUcNUVq+p48e3EPvwoKlUAHad8n0a945wSyE?= =?us-ascii?Q?TiBbYpfag7cuofHINGG5WwxQt5LOSLEH48y2KaHGgSMBWkRybrSe+74B1Phx?= =?us-ascii?Q?jhLiRWWPCypuTTLdjIawCYEJVg+QzufHy2ZhwR7Sz4dauQavCdgf3XxNmBIB?= =?us-ascii?Q?5WykQ+ELY0ahUZBZlXY8F5l82vjdgZWQKOSTwhEhi9u4ICAnV1CGhsIMbYZ1?= =?us-ascii?Q?GfIMIWNSyNxjgvq082EytpGolv0/Eh/hDk9OCaio2HoZ9o6jXRttyj87B45H?= =?us-ascii?Q?Ms2q233uoYdU7GRztEUMhw5MEugkvgTrY1N7MXPF5Vge2boq+5MLcsX7An8z?= =?us-ascii?Q?h4khzD9dbTWsNBEgmu0W64UPtqCJpmzcDHPuSENlACBpXcdQ2qUSEgjOSjSQ?= =?us-ascii?Q?45qFsGAZkKXubjqLEsGWuSO+cuj07+FAD2iyayPncp2+K1puBWh90kt7i6Sn?= =?us-ascii?Q?wsbSRbbC6vZtaVIsBQVjix/igAckxl5u7+kKkqNx2jSi5hU5yyc76DkndXK9?= =?us-ascii?Q?MglVFYJYxDfHtOSgZLWERMC/VOd61LWkWJQt0EbGibh20IuF2o+Q8WjfGPB0?= =?us-ascii?Q?id8x9dsIaD7aUVccL7tn2dgFGndcl4krEq6mjnAKEp7DQrxq+iTWZEdpNUYD?= =?us-ascii?Q?0AcLRz6z3q9FD/XTb38fPOs5L24mUpzdeUAR/KOBIG5dKtqZ0es0Na9i3yv9?= =?us-ascii?Q?7Ml816Z9e36zjUIuc2SMdjBYfJVHaXpPJ8cI17dgjp2SAM/2CjSKh1HWU65H?= =?us-ascii?Q?TEJ7tgfM/kOnQe+QKtiOxULc0TBvKojKoVbOeF1n7Zy8hRdkRfe5cb1fL9WB?= =?us-ascii?Q?2B3KPnRuo5JjiQVButw6HksDxL9jBFWPRkFs0fNXSGqlb2Bd9ZAz0X1HD5Hv?= =?us-ascii?Q?osz9tuwIyWUPWzBIaQNNPGYzmzqR6vsMMSL6QSp4rXbZjXlt3bQyXlNaQwnV?= =?us-ascii?Q?1chNL0pcXj5Ifuj+oJ3tPwuQbwCZjd4uDn1VZGv/r+TGYMPLp3BfI7hgO+2d?= =?us-ascii?Q?6XsvpFNDIbymB5DV8mLmVPY1tOEbC77M/vS5/ooMMQdHMKBvLi9u6IM/L//L?= =?us-ascii?Q?AgYp7Ph+3UaPF4ufSiT3AgTH9qjHrZZXiOAZGbYbQi3j7qgAOP7f5ecQj2XE?= =?us-ascii?Q?CxGKf9mS5CSRx+KoYpubLNfQZNdwbumnemZ9fe1QxvfQfMBcxe2YC3WytuQ/?= =?us-ascii?Q?JIYRv3jZfViQnIaKLEy9Jq8T3ZxBxYmI+mNqgONG7EYR4382rbd8XTjA2J4B?= =?us-ascii?Q?bn6NzsR3Qg=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR11MB3504.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b46f57f-d630-4b09-c140-08da3c9171c3 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2022 07:54:20.2769 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: GaPLyi0MM/H/S+TSawDytFyEGsPjXHv9OtIelIqvnk6h2Y2/eIcao+tV8x6lHDy2PvFvN+Z84wlE0Ah02lNYTA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB5596 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pei, Andy > Sent: Wednesday, May 18, 2022 8:14 PM > To: dev@dpdk.org > Cc: Xia, Chenbo ; maxime.coquelin@redhat.com; Cao, > Gang ; Liu, Changpeng ; Xu, > Rosen ; Xiao, QimaiX > Subject: [PATCH v8 12/13] vdpa/ifc: add interrupt handling for config > space >=20 > Create a thread to poll and relay config space change interrupt. > Use VHOST_USER_SLAVE_CONFIG_CHANGE_MSG to inform QEMU. >=20 > Signed-off-by: Andy Pei > --- > drivers/vdpa/ifc/ifcvf_vdpa.c | 118 > +++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 117 insertions(+), 1 deletion(-) >=20 > diff --git a/drivers/vdpa/ifc/ifcvf_vdpa.c b/drivers/vdpa/ifc/ifcvf_vdpa.= c > index 376a1af..8a49622 100644 > --- a/drivers/vdpa/ifc/ifcvf_vdpa.c > +++ b/drivers/vdpa/ifc/ifcvf_vdpa.c > @@ -53,7 +53,9 @@ struct ifcvf_internal { > int vfio_group_fd; > int vfio_dev_fd; > pthread_t tid; /* thread for notify relay */ > + pthread_t intr_tid; /* thread for config space change interrupt > relay */ > int epfd; > + int csc_epfd; > int vid; > struct rte_vdpa_device *vdev; > uint16_t max_queues; > @@ -566,6 +568,111 @@ struct rte_vdpa_dev_info { > return 0; > } >=20 > +static void > +virtio_interrupt_handler(struct ifcvf_internal *internal) > +{ > + int vid =3D internal->vid; > + int ret; > + > + ret =3D rte_vhost_slave_config_change(vid, 1); > + if (ret) > + DRV_LOG(ERR, "failed to notify the guest about configuration > space change."); > +} > + > +static void * > +intr_relay(void *arg) > +{ > + struct ifcvf_internal *internal =3D (struct ifcvf_internal *)arg; > + struct epoll_event csc_event; > + struct epoll_event ev; > + uint64_t buf; > + int nbytes; > + int csc_epfd, csc_val =3D 0; > + > + csc_epfd =3D epoll_create(1); > + if (csc_epfd < 0) { > + DRV_LOG(ERR, "failed to create epoll for config space > change."); > + return NULL; > + } > + > + ev.events =3D EPOLLIN | EPOLLPRI | EPOLLRDHUP | EPOLLHUP; > + ev.data.fd =3D rte_intr_fd_get(internal->pdev->intr_handle); > + if (epoll_ctl(csc_epfd, EPOLL_CTL_ADD, > + rte_intr_fd_get(internal->pdev->intr_handle), &ev) < 0) { > + DRV_LOG(ERR, "epoll add error: %s", strerror(errno)); > + return NULL; > + } > + > + internal->csc_epfd =3D csc_epfd; > + > + for (;;) { > + csc_val =3D epoll_wait(csc_epfd, &csc_event, 1, -1); > + if (csc_val < 0) { > + if (errno =3D=3D EINTR) > + continue; > + DRV_LOG(ERR, "epoll_wait return fail."); > + return NULL; > + } else if (csc_val =3D=3D 0) { > + continue; > + } else { > + /* csc_val > 0 */ > + nbytes =3D read(csc_event.data.fd, &buf, 8); > + if (nbytes < 0) { > + if (errno =3D=3D EINTR || > + errno =3D=3D EWOULDBLOCK || > + errno =3D=3D EAGAIN) > + continue; > + DRV_LOG(ERR, "Error reading from file > descriptor %d: %s\n", > + csc_event.data.fd, > + strerror(errno)); > + return NULL; > + } else if (nbytes =3D=3D 0) { > + DRV_LOG(ERR, "Read nothing from file > descriptor %d\n", > + csc_event.data.fd); > + continue; > + } else { > + virtio_interrupt_handler(internal); > + } > + } > + } > + > + return NULL; > +} I think we should not assume unset_intr_relay will help us close epfd when Error happens, so just close it when there's some error. > + > +static int > +setup_intr_relay(struct ifcvf_internal *internal) > +{ > + char name[THREAD_NAME_LEN]; > + int ret; > + > + snprintf(name, sizeof(name), "ifc-intr-%d", internal->vid); > + ret =3D rte_ctrl_thread_create(&internal->intr_tid, name, NULL, > + intr_relay, (void *)internal); > + if (ret) { > + DRV_LOG(ERR, "failed to create notify relay pthread."); > + return -1; > + } > + return 0; > +} > + > +static int > +unset_intr_relay(struct ifcvf_internal *internal) > +{ > + void *status; > + > + if (internal->intr_tid) { > + pthread_cancel(internal->intr_tid); > + pthread_join(internal->intr_tid, &status); > + } > + internal->intr_tid =3D 0; > + > + if (internal->csc_epfd >=3D 0) > + close(internal->csc_epfd); > + internal->csc_epfd =3D -1; > + > + return 0; > +} It will always return 0, so return type should be void > + > static int > update_datapath(struct ifcvf_internal *internal) > { > @@ -592,10 +699,16 @@ struct rte_vdpa_dev_info { > if (ret) > goto err; >=20 > + ret =3D setup_intr_relay(internal); > + if (ret) > + goto err; > + > rte_atomic32_set(&internal->running, 1); > } else if (rte_atomic32_read(&internal->running) && > (!rte_atomic32_read(&internal->started) || > !rte_atomic32_read(&internal->dev_attached))) { > + ret =3D unset_intr_relay(internal); This will be changed accordingly. Thanks, Chenbo > + > ret =3D unset_notify_relay(internal); > if (ret) > goto err; > @@ -812,7 +925,7 @@ struct rte_vdpa_dev_info { > if (nfds < 0) { > if (errno =3D=3D EINTR) > continue; > - DRV_LOG(ERR, "epoll_wait return fail\n"); > + DRV_LOG(ERR, "epoll_wait return fail."); > return NULL; > } >=20 > @@ -888,6 +1001,9 @@ struct rte_vdpa_dev_info { > /* stop the direct IO data path */ > unset_notify_relay(internal); > vdpa_ifcvf_stop(internal); > + > + unset_intr_relay(internal); > + > vdpa_disable_vfio_intr(internal); >=20 > ret =3D rte_vhost_host_notifier_ctrl(vid, RTE_VHOST_QUEUE_ALL, false); > -- > 1.8.3.1