From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7C9FAA052B; Thu, 30 Jul 2020 07:48:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E836AE07; Thu, 30 Jul 2020 07:48:46 +0200 (CEST) Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 182EBA69 for ; Thu, 30 Jul 2020 07:48:44 +0200 (CEST) IronPort-SDR: Cgx9sxpBPdtGD+ZeLlw1RLsYb13nJpQOCHXPui9DI56uTLYBK8EoqRMP+fG3CR6TdFEIWa+hCm VMp0tfINrR+Q== X-IronPort-AV: E=McAfee;i="6000,8403,9697"; a="149005308" X-IronPort-AV: E=Sophos;i="5.75,413,1589266800"; d="scan'208";a="149005308" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jul 2020 22:48:43 -0700 IronPort-SDR: b2HxIY3TvyGsKurbhuWBVW5Df+4PGP3NjlEVf5cwtaMnPpJK/zF59uWzi60dVR21Lt9OE1Ritr fOd2dbxHVWuQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,413,1589266800"; d="scan'208";a="272835824" Received: from orsmsx601.amr.corp.intel.com ([10.22.229.14]) by fmsmga007.fm.intel.com with ESMTP; 29 Jul 2020 22:48:43 -0700 Received: from orsmsx605.amr.corp.intel.com (10.22.229.18) by ORSMSX601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 29 Jul 2020 22:48:43 -0700 Received: from ORSEDG002.ED.cps.intel.com (10.7.248.5) by orsmsx605.amr.corp.intel.com (10.22.229.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Wed, 29 Jul 2020 22:48:43 -0700 Received: from NAM04-CO1-obe.outbound.protection.outlook.com (104.47.45.50) by edgegateway.intel.com (134.134.137.101) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 29 Jul 2020 22:48:41 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ETwEAxVD8cwTOqAsDXhl1ZBhA0AMFYJxyR9p7wSqaHpJrUkbOC5tc6WE7f/o3/TUe1FFhVBiwWcaFJmY2VOFl2P4Xd0eseuNAyxPgx5RjCvAKuXSVyAhHbH/I1zU0afcuBjDZm47Ea+jbnpV8qisodDl8SItWZxzYo/aXNE40V/XiFGxg7B2MW4NTa1ky6cSvNlIQKr+nuI28PscaZB8yeTZ5762Jo+QtaA0Ob4zNkuuuUlvWFNJHLiU8FiCN83IyMVGMiz4g12wrCrqYcWjeDY/fHPFUTNw3h11hoKVa4NLQnx/T7L3BThteFK05twd3b69YqbgwxRXgu4+WQcdEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=znQ2Ba4hlrj5LevEAD3CjXoacIJdP6W+xI7XFh/rmBg=; b=IRdPKCQV/RTquT0J7S568ODgXCvDkc7ia6kSgARN/KDBQ6ZGB5teNv5KUBlzbE8wqKKLCIsDn5JXYeysIS2ReGvD3y2qKCKDifXl2GsuZSIbbQ+UpGrQDLSAXpibkoA7ggJHizcZdEEe/hWXScZHlxbUSl44eKoSrI6uKOgE91pJPNA16k3xCgMyYhBKqWmrVMqRAqGA6CdlJguh1A6K1qRtEIi4CnZjjYhgVyn6XdV7DaQVMwGhv2m5XxOqW7vNjgQp2kxccmSzGCMNOWnfr2FYNcpY408o57FLguVn4k1g9Oku6H36xJO9jL/jiofkuaIqXIJETi+rfaDsbbKPSQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=znQ2Ba4hlrj5LevEAD3CjXoacIJdP6W+xI7XFh/rmBg=; b=CXIWzLpLgE78Q30zugIrbpPcsyP3Y2I7PCTrvIXl3COITymnpQQjUShgpSyTFKKaCkqo91Ni7hTRBBlPOkg9y89rWmf9o1dr2Fqm71OsmS1QAhFyaqeGoKuvGAn2w09sq00xVAWJoXMPCCHlKmCN5kj8QneSO1Dpvwo9f1liMzQ= Received: from MWHPR1101MB2254.namprd11.prod.outlook.com (2603:10b6:301:58::11) by MWHPR11MB1389.namprd11.prod.outlook.com (2603:10b6:300:26::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3239.17; Thu, 30 Jul 2020 05:48:34 +0000 Received: from MWHPR1101MB2254.namprd11.prod.outlook.com ([fe80::206d:f3f3:b571:41ee]) by MWHPR1101MB2254.namprd11.prod.outlook.com ([fe80::206d:f3f3:b571:41ee%11]) with mapi id 15.20.3239.019; Thu, 30 Jul 2020 05:48:34 +0000 From: "Xiao, QimaiX" To: "Xia, Chenbo" , Maxime Coquelin , "dev@dpdk.org" , "matan@mellanox.com" , "Liu, Yong" , "Wang, Yinan" CC: "thomas@monjalon.net" , "Yigit, Ferruh" , "david.marchand@redhat.com" Thread-Topic: [PATCH v4 3/3] net/vhost: fix interrupt mode Thread-Index: AQHWZbM/hfgmxLkNYkmvKQ8pU8Xtb6kfnmfg Date: Thu, 30 Jul 2020 05:48:33 +0000 Message-ID: References: <20200729133626.237098-1-maxime.coquelin@redhat.com> <20200729133626.237098-4-maxime.coquelin@redhat.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.55.46.46] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 863aff9d-2eda-451d-30ca-08d8344c327d x-ms-traffictypediagnostic: MWHPR11MB1389: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 7p7C1c76If9SX7tN0TrKUewX6FIsyeRQnzgidYGFePrKnmaOT/FHyvn5imQhoXYZX6rmcX1jziySETZwnJep8n6d6S3j4Eug7u798WMioM2E8oZhKQFEPlJx4OygjY1H5LrKeH9czIz/Qi28tkbtot2OeJEkR86J59WCXPALmqgRjCoiDwzQh57HqhclldJYTZqwtIJ63876V8oNPvK9vB0vbx+AD2H/PN9g7cnsfA26ZtEcRrrs0MEYzs0Vc4Pc0+9yZceoLMICklb+bNq9AXfZpLx3pQU5BVTJBUo+elt/c5nqWeg/3e3dGfMHXm96yDZiDa0/Fgtiqv5J2jStAw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR1101MB2254.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(376002)(136003)(366004)(396003)(346002)(39860400002)(316002)(66446008)(66476007)(66556008)(64756008)(52536014)(66946007)(76116006)(5660300002)(71200400001)(7696005)(186003)(2906002)(33656002)(110136005)(54906003)(478600001)(26005)(6506007)(53546011)(8676002)(8936002)(86362001)(9686003)(83380400001)(55016002)(6636002)(4326008); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: +QUt6TbwFQ6NcR4q2jkPQ9ARSdWzrLPH9KSYyMWVf/sxwyUxgLpSmDR7zxO14wcC702L+VjRFSlE7X5wOOEAtmT59TEoL1jRNThX+ILkUnUo8ROqhON3wBsR23p/CP0c8PeKPioY0V33QN2o6LUiRzvesENlTTXZSr+7uQNbI0xhgsLLVPMddr5QJuFWCJZT33q+uDrWyIfd7qShfUhrD4O1eGrUfYqerY0/dgp71/K+6dpzv8DaIoH5mfYqN+sfwObzvLWu4owmr2oNOOYme9/hUeep5focFL98XXxsGMA5hpiJPQ/PrJm7DPNNFb1yqmR7hNeI2+cznUGFOpFBMYX8Gxj6KKYR4qaBS+C9Oba+43YhyRApMCDZQ6DcnAjB7mhzLdf9s9VBivGwhjXRAHkQi6J1ANLQVol7mTNcMGzEkri2nYjU49LJnSlnWLQo3o2HTJWEqmZWbX/eVUeuGS8etxjFQn5oRWdRrlwGXeQLoVumcbpXUTrq/4TIpJy7Bx1bCcX0Ifgf/TPH5XlLIhgxoaTMaekcazJF+tGSBXsJJWEo0gwZEXOHaU8bjHG/GmsGkS5Qe6Vtit4/0EwvZsqGMKbkwaNvkieV5veywHFG2Ut4VRxOz1FKXuzjaxibgE9RDNvNY86E9FfW11aExg== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2254.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 863aff9d-2eda-451d-30ca-08d8344c327d X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Jul 2020 05:48:33.9470 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ZIrPc3GsFT1wpH6jAGrRW5pUoYxLs5ZX2C/H0l+3RACfc76lAqEr8epYaD+iqGn5SyTHoqaOCljO7IZrYmN1rA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR11MB1389 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v4 3/3] net/vhost: fix interrupt mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Tested-by: Xiao Qimai Regards, Xiao Qimai > -----Original Message----- > From: dev On Behalf Of Xia, Chenbo > Sent: Wednesday, July 29, 2020 10:19 PM > To: Maxime Coquelin ; dev@dpdk.org; > matan@mellanox.com; Liu, Yong ; Wang, Yinan > > Cc: thomas@monjalon.net; Yigit, Ferruh ; > david.marchand@redhat.com > Subject: Re: [dpdk-dev] [PATCH v4 3/3] net/vhost: fix interrupt mode >=20 >=20 > > -----Original Message----- > > From: Maxime Coquelin > > Sent: Wednesday, July 29, 2020 9:36 PM > > To: dev@dpdk.org; matan@mellanox.com; Xia, Chenbo > > ; Liu, Yong ; Wang, Yinan > > > > Cc: thomas@monjalon.net; Yigit, Ferruh ; > > david.marchand@redhat.com; Maxime Coquelin > > > > Subject: [PATCH v4 3/3] net/vhost: fix interrupt mode > > > > At .new_device() time, only the first vring pair is now ready, other > > vrings are configured later. > > > > Problem is that when application will setup and enable interrupts, > > only the first queue pair Rx interrupt will be enabled. > > > > This patches fixes the issue by setting the number of max interrupts > > to the number of Rx queues that will be later initialized. Then, as > > soon as a Rx vring is ready and interrupt enabled by the application, > > it removes the corresponding uninitialized epoll event, and installs a = new > one with the valid FD. > > > > Fixes: 604052ae5395 ("net/vhost: support queue update") > > > > Signed-off-by: Maxime Coquelin > > --- > > drivers/net/vhost/rte_eth_vhost.c | 91 > > +++++++++++++++++++++++++++---- > > 1 file changed, 81 insertions(+), 10 deletions(-) > > > > diff --git a/drivers/net/vhost/rte_eth_vhost.c > > b/drivers/net/vhost/rte_eth_vhost.c > > index 951929c663..e55278af69 100644 > > --- a/drivers/net/vhost/rte_eth_vhost.c > > +++ b/drivers/net/vhost/rte_eth_vhost.c > > @@ -5,6 +5,7 @@ > > #include > > #include > > #include > > +#include > > > > #include > > #include > > @@ -95,6 +96,8 @@ struct vhost_queue { > > uint16_t port; > > uint16_t virtqueue_id; > > struct vhost_stats stats; > > + int intr_enable; > > + rte_spinlock_t intr_lock; > > }; > > > > struct pmd_internal { > > @@ -524,12 +527,58 @@ find_internal_resource(char *ifname) > > return list; > > } > > > > +static int > > +eth_vhost_update_intr(struct rte_eth_dev *eth_dev, uint16_t rxq_idx) { > > + struct rte_intr_handle *handle =3D eth_dev->intr_handle; > > + struct rte_epoll_event rev; > > + int epfd, ret; > > + > > + if (!handle) > > + return 0; > > + > > + if (handle->efds[rxq_idx] =3D=3D handle->elist[rxq_idx].fd) > > + return 0; > > + > > + VHOST_LOG(INFO, "kickfd for rxq-%d was changed, updating > > handler.\n", > > + rxq_idx); > > + > > + if (handle->elist[rxq_idx].fd !=3D -1) > > + VHOST_LOG(ERR, "Unexpected previous kickfd value > (Got %d, > > expected -1).\n", > > + handle->elist[rxq_idx].fd); > > + > > + /* > > + * First remove invalid epoll event, and then install > > + * the new one. May be solved with a proper API in the > > + * future. > > + */ > > + epfd =3D handle->elist[rxq_idx].epfd; > > + rev =3D handle->elist[rxq_idx]; > > + ret =3D rte_epoll_ctl(epfd, EPOLL_CTL_DEL, rev.fd, > > + &handle->elist[rxq_idx]); > > + if (ret) { > > + VHOST_LOG(ERR, "Delete epoll event failed.\n"); > > + return ret; > > + } > > + > > + rev.fd =3D handle->efds[rxq_idx]; > > + handle->elist[rxq_idx] =3D rev; > > + ret =3D rte_epoll_ctl(epfd, EPOLL_CTL_ADD, rev.fd, > > + &handle->elist[rxq_idx]); > > + if (ret) { > > + VHOST_LOG(ERR, "Add epoll event failed.\n"); > > + return ret; > > + } > > + > > + return 0; > > +} > > + > > static int > > eth_rxq_intr_enable(struct rte_eth_dev *dev, uint16_t qid) { > > struct rte_vhost_vring vring; > > struct vhost_queue *vq; > > - int ret =3D 0; > > + int old_intr_enable, ret =3D 0; > > > > vq =3D dev->data->rx_queues[qid]; > > if (!vq) { > > @@ -537,6 +586,18 @@ eth_rxq_intr_enable(struct rte_eth_dev *dev, > > uint16_t > > qid) > > return -1; > > } > > > > + rte_spinlock_lock(&vq->intr_lock); > > + old_intr_enable =3D vq->intr_enable; > > + vq->intr_enable =3D 1; > > + ret =3D eth_vhost_update_intr(dev, qid); > > + rte_spinlock_unlock(&vq->intr_lock); > > + > > + if (ret < 0) { > > + VHOST_LOG(ERR, "Failed to update rxq%d's intr\n", qid); > > + vq->intr_enable =3D old_intr_enable; > > + return ret; > > + } > > + > > ret =3D rte_vhost_get_vhost_vring(vq->vid, (qid << 1) + 1, &vring); > > if (ret < 0) { > > VHOST_LOG(ERR, "Failed to get rxq%d's vring\n", qid); @@ - > > 571,6 +632,8 @@ eth_rxq_intr_disable(struct rte_eth_dev *dev, uint16_t > qid) > > rte_vhost_enable_guest_notification(vq->vid, (qid << 1) + 1, 0); > > rte_wmb(); > > > > + vq->intr_enable =3D 0; > > + > > return 0; > > } > > > > @@ -593,7 +656,6 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) { > > struct rte_vhost_vring vring; > > struct vhost_queue *vq; > > - int count =3D 0; > > int nb_rxq =3D dev->data->nb_rx_queues; > > int i; > > int ret; > > @@ -623,6 +685,8 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) > > > > VHOST_LOG(INFO, "Prepare intr vec\n"); > > for (i =3D 0; i < nb_rxq; i++) { > > + dev->intr_handle->intr_vec[i] =3D > RTE_INTR_VEC_RXTX_OFFSET + > > i; > > + dev->intr_handle->efds[i] =3D -1; > > vq =3D dev->data->rx_queues[i]; > > if (!vq) { > > VHOST_LOG(INFO, "rxq-%d not setup yet, skip!\n", i); > @@ -641,14 > > +705,12 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) > > "rxq-%d's kickfd is invalid, skip!\n", i); > > continue; > > } > > - dev->intr_handle->intr_vec[i] =3D > RTE_INTR_VEC_RXTX_OFFSET + > > i; > > dev->intr_handle->efds[i] =3D vring.kickfd; > > - count++; > > VHOST_LOG(INFO, "Installed intr vec for rxq-%d\n", i); > > } > > > > - dev->intr_handle->nb_efd =3D count; > > - dev->intr_handle->max_intr =3D count + 1; > > + dev->intr_handle->nb_efd =3D nb_rxq; > > + dev->intr_handle->max_intr =3D nb_rxq + 1; > > dev->intr_handle->type =3D RTE_INTR_HANDLE_VDEV; > > > > return 0; > > @@ -835,6 +897,7 @@ vring_conf_update(int vid, struct rte_eth_dev > > *eth_dev, uint16_t vring_id) { > > struct rte_eth_conf *dev_conf =3D ð_dev->data->dev_conf; > > struct pmd_internal *internal =3D eth_dev->data->dev_private; > > + struct vhost_queue *vq; > > struct rte_vhost_vring vring; > > int rx_idx =3D vring_id % 2 ? (vring_id - 1) >> 1 : -1; > > int ret =3D 0; > > @@ -853,12 +916,18 @@ vring_conf_update(int vid, struct rte_eth_dev > > *eth_dev, uint16_t vring_id) > > vring_id); > > return ret; > > } > > + eth_dev->intr_handle->efds[rx_idx] =3D vring.kickfd; > > > > - if (vring.kickfd !=3D eth_dev->intr_handle->efds[rx_idx]) { > > - VHOST_LOG(INFO, "kickfd for rxq-%d was > changed.\n", > > - rx_idx); > > - eth_dev->intr_handle->efds[rx_idx] =3D vring.kickfd; > > + vq =3D eth_dev->data->rx_queues[rx_idx]; > > + if (!vq) { > > + VHOST_LOG(ERR, "rxq%d is not setup yet\n", rx_idx); > > + return -1; > > } > > + > > + rte_spinlock_lock(&vq->intr_lock); > > + if (vq->intr_enable) > > + ret =3D eth_vhost_update_intr(eth_dev, rx_idx); > > + rte_spinlock_unlock(&vq->intr_lock); > > } > > > > return ret; > > @@ -1152,6 +1221,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, > > uint16_t rx_queue_id, > > > > vq->mb_pool =3D mb_pool; > > vq->virtqueue_id =3D rx_queue_id * VIRTIO_QNUM + VIRTIO_TXQ; > > + rte_spinlock_init(&vq->intr_lock); > > dev->data->rx_queues[rx_queue_id] =3D vq; > > > > return 0; > > @@ -1173,6 +1243,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, > > uint16_t tx_queue_id, > > } > > > > vq->virtqueue_id =3D tx_queue_id * VIRTIO_QNUM + VIRTIO_RXQ; > > + rte_spinlock_init(&vq->intr_lock); > > dev->data->tx_queues[tx_queue_id] =3D vq; > > > > return 0; > > -- > > 2.26.2 >=20 > Reviewed-by: Chenbo Xia