From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 10EB5A052B; Wed, 29 Jul 2020 16:18:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 982A84C98; Wed, 29 Jul 2020 16:18:47 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 3C44FA3 for ; Wed, 29 Jul 2020 16:18:46 +0200 (CEST) IronPort-SDR: tSy2ih754oLr2ebZoAN9kZZFERHj5lASp8Spx4s3+WBs953fAsDhyZjc9JWo1cuxbk0axZ03pP zsslynDe4I/A== X-IronPort-AV: E=McAfee;i="6000,8403,9696"; a="151399377" X-IronPort-AV: E=Sophos;i="5.75,410,1589266800"; d="scan'208";a="151399377" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jul 2020 07:18:45 -0700 IronPort-SDR: YVeH0+rTx4dElmOc25P0aRrjTi7Qwu6lko2shCgyRd/UTnAZdg7JXkVonJqMf1pqamoKuCJ8Oj HFKxz3B03XWA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,410,1589266800"; d="scan'208";a="330414805" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by orsmga007.jf.intel.com with ESMTP; 29 Jul 2020 07:18:44 -0700 Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Wed, 29 Jul 2020 07:18:44 -0700 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Wed, 29 Jul 2020 07:18:44 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.177) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 29 Jul 2020 07:18:43 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jBUsbiUPOMxmklTHn8EDjtQlgS8WIPRicrnngBucx1eRYO1Sh+0eS0rqhrPHCGzN3fo26Lkuebh7HWUJjRw/LwAQq1/qKHHIGioEDhWjGBul0snKAWgUJ1XuiS2NY0eJOowxGgeuUDdsGzmcf7I1mOzOQ4LdC/qooBOOGJT56R3L1eTIhGHWutIxA//4ST3RDjJ+aYU2JPkdnzi3p1jJQWd2e6q8ioHChL/GgXj2yPqFqd3nbEJ31EsTTcnr+xKGzlXiPvMai2egn0RB8tsD5TRyhG3rZkNAWTj5lkvzjTUrMEqyMsjb38Vozixow3FHkBpacMrQZWPrmYHEjKEb0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Q5dLWIATNOzlsFP3GswnACt2sT9wGYGs5laVRw7Glo=; b=WGhx8cxzNNupE/yc5ZsydQKGWvJHZHMsge4+rz1cqbQgmUVCHP1Qqk/ZIXhsEyMJU2z7AmWo63XGkgR6zcJsIptAGNIlhVNSQQVak7JfbhN8MAooH/bSrwMTlDQc50GdeVrCNQmdA1qHphi/P5okWdxy02XwW3hM6JH52BWAtDKM2I8tkP5MYSfO6IhHm8cR82unacb1/vlRNacNlpqCxduaqIlNO9oNqE0A561+2DQga13O3fPAjtVPjn+dxfQFWtSH64G/bg+jH7PzVwoyl/Jlzm0u6TECSOZPDLs3jrJ5+5wGdxJwyesK1yEF8e/2a1sXXMTMX0B6EUA/jeEnrg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Q5dLWIATNOzlsFP3GswnACt2sT9wGYGs5laVRw7Glo=; b=u1rQ3dFtRQNpOj8cwYgLX22SoREsmSML3D2Stg/DhMZbg8Y1hE9aQapUbIHFkvxcPSdyRFCKU2VwugFTLP+oHUrnut7p24+ZhbIfwuiwnWX64DNo5SJZhA80dsbkYtlPxgg4XBJ6xQOsN2zgBuRscBuiU1h7vuqxDzSE32C20LY= Received: from MN2PR11MB4063.namprd11.prod.outlook.com (2603:10b6:208:13f::22) by MN2PR11MB3696.namprd11.prod.outlook.com (2603:10b6:208:f6::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3239.17; Wed, 29 Jul 2020 14:18:38 +0000 Received: from MN2PR11MB4063.namprd11.prod.outlook.com ([fe80::b898:36f5:61cb:42ca]) by MN2PR11MB4063.namprd11.prod.outlook.com ([fe80::b898:36f5:61cb:42ca%7]) with mapi id 15.20.3216.034; Wed, 29 Jul 2020 14:18:38 +0000 From: "Xia, Chenbo" To: Maxime Coquelin , "dev@dpdk.org" , "matan@mellanox.com" , "Liu, Yong" , "Wang, Yinan" CC: "thomas@monjalon.net" , "Yigit, Ferruh" , "david.marchand@redhat.com" Thread-Topic: [PATCH v4 3/3] net/vhost: fix interrupt mode Thread-Index: AQHWZa1WzLb3uJe9GEWD8zGSnIikqKkemq4g Date: Wed, 29 Jul 2020 14:18:38 +0000 Message-ID: References: <20200729133626.237098-1-maxime.coquelin@redhat.com> <20200729133626.237098-4-maxime.coquelin@redhat.com> In-Reply-To: <20200729133626.237098-4-maxime.coquelin@redhat.com> Accept-Language: en-US, zh-CN Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 authentication-results: redhat.com; dkim=none (message not signed) header.d=none;redhat.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [192.198.147.205] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 11f52a99-b7d2-4cf3-bba3-08d833ca49a9 x-ms-traffictypediagnostic: MN2PR11MB3696: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5t+KpBHXY3XzmIPX0T+LxbrF7IVttfAsgl9EKqEn9+9591OpuIZPdnwJSyeUj8kee/q03WbD+iCpwJBWfEhbrG5YYC2krEHRUyRbxsW6Dif4oXm1lp+RaNcrGA1eYM8xSSN96I93UW9OMIDjSzGyJwQ0pxV/eSPu79FrHHXbbnxYWM75FGUBclZ4PX7JczUxXK66fYAUHRM1SDA/jt0dEaIS++9lG/7wTlJR57WyrrRTnU6U31NxSxcQ0mBJZrrhubHKxoMfnnpEBn4Ao1nZSZ58pGYDRrhtilmdb4KHe2DGYH4ali3LeJBJrevqKmLPnwHzVQbsC/TP8oBDHnTGmg== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR11MB4063.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(376002)(366004)(136003)(396003)(346002)(39860400002)(53546011)(7696005)(66946007)(6506007)(64756008)(66446008)(66556008)(66476007)(5660300002)(76116006)(8676002)(6636002)(33656002)(4326008)(316002)(26005)(8936002)(186003)(52536014)(478600001)(2906002)(83380400001)(55016002)(54906003)(9686003)(71200400001)(110136005)(86362001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: 5XJFzn6iukfKaTWKxD/SypGhKDYuIQjt9KJ+qOTujpiJWXLW5hrJDpCzuhiZf1X/SLcQiw4YYID24vZ3Jo9azWsggXkKdCOzNSQ1pQqaMvc/cSpgDhOQnD0NTxs4qkKUAUjzMgOIdzFXHsC1v++RzTeTbbKUv968dpGGYAIFAY7MRkmHJ/so6lnFbgFkhkBqdn1AtqDpL7UmUx4o2LLdSe43/8de57oUGF6LQnrNE1jw2pfpYxpBVxqAprIPJa6cc6ePZU8zE2rfNqxVIn/3PqJettWfzPbQFiumivlFn75IvjjXZxkHIa3naQkRfwpwwGXQPj/er2eDMwmAzgec0MC2Spi7edlbLVssq7ikHvW1bfHVW8rHjunOcz3/oRZ1fYYQp8kxWVK2lXvg7tRMuIma1Ti391n3sGRnoNTvdxZKq1MWt4XTSJG21l4hJalPxvHIwaElmOYjW7NQ5ZGdJbQQ9/xk6kvF/dT/kPyTIVlKNZYOLnMbXQkPLdzxJohZGxPRe0o72cz4lmhYuXuEYtXIsXcObPwWxNSaSlsZKV3C3+zHdwDWFNT+v+aj1w2tvTUnMooVB8wKimwn2fguoAJsrdu6GES8aKg178UdC98wwGDEmchDjZcyhGf+tVC7nd5gnwSkbqW9pJesAPuftg== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: MN2PR11MB4063.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 11f52a99-b7d2-4cf3-bba3-08d833ca49a9 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jul 2020 14:18:38.2677 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: WzISpTf79JziGKW8lb0KIke9REZHi4f5tWBtqn1ifZW+mSYISpnmHHQkfNGFDmY++tz1VSyr/oEqwexEYsv5Vg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB3696 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH v4 3/3] net/vhost: fix interrupt mode X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Maxime Coquelin > Sent: Wednesday, July 29, 2020 9:36 PM > To: dev@dpdk.org; matan@mellanox.com; Xia, Chenbo > ; Liu, Yong ; Wang, Yinan > > Cc: thomas@monjalon.net; Yigit, Ferruh ; > david.marchand@redhat.com; Maxime Coquelin > > Subject: [PATCH v4 3/3] net/vhost: fix interrupt mode >=20 > At .new_device() time, only the first vring pair is now ready, other vrin= gs are > configured later. >=20 > Problem is that when application will setup and enable interrupts, only t= he first > queue pair Rx interrupt will be enabled. >=20 > This patches fixes the issue by setting the number of max interrupts to t= he > number of Rx queues that will be later initialized. Then, as soon as a Rx= vring is > ready and interrupt enabled by the application, it removes the correspond= ing > uninitialized epoll event, and installs a new one with the valid FD. >=20 > Fixes: 604052ae5395 ("net/vhost: support queue update") >=20 > Signed-off-by: Maxime Coquelin > --- > drivers/net/vhost/rte_eth_vhost.c | 91 +++++++++++++++++++++++++++---- > 1 file changed, 81 insertions(+), 10 deletions(-) >=20 > diff --git a/drivers/net/vhost/rte_eth_vhost.c > b/drivers/net/vhost/rte_eth_vhost.c > index 951929c663..e55278af69 100644 > --- a/drivers/net/vhost/rte_eth_vhost.c > +++ b/drivers/net/vhost/rte_eth_vhost.c > @@ -5,6 +5,7 @@ > #include > #include > #include > +#include >=20 > #include > #include > @@ -95,6 +96,8 @@ struct vhost_queue { > uint16_t port; > uint16_t virtqueue_id; > struct vhost_stats stats; > + int intr_enable; > + rte_spinlock_t intr_lock; > }; >=20 > struct pmd_internal { > @@ -524,12 +527,58 @@ find_internal_resource(char *ifname) > return list; > } >=20 > +static int > +eth_vhost_update_intr(struct rte_eth_dev *eth_dev, uint16_t rxq_idx) { > + struct rte_intr_handle *handle =3D eth_dev->intr_handle; > + struct rte_epoll_event rev; > + int epfd, ret; > + > + if (!handle) > + return 0; > + > + if (handle->efds[rxq_idx] =3D=3D handle->elist[rxq_idx].fd) > + return 0; > + > + VHOST_LOG(INFO, "kickfd for rxq-%d was changed, updating > handler.\n", > + rxq_idx); > + > + if (handle->elist[rxq_idx].fd !=3D -1) > + VHOST_LOG(ERR, "Unexpected previous kickfd value (Got %d, > expected -1).\n", > + handle->elist[rxq_idx].fd); > + > + /* > + * First remove invalid epoll event, and then install > + * the new one. May be solved with a proper API in the > + * future. > + */ > + epfd =3D handle->elist[rxq_idx].epfd; > + rev =3D handle->elist[rxq_idx]; > + ret =3D rte_epoll_ctl(epfd, EPOLL_CTL_DEL, rev.fd, > + &handle->elist[rxq_idx]); > + if (ret) { > + VHOST_LOG(ERR, "Delete epoll event failed.\n"); > + return ret; > + } > + > + rev.fd =3D handle->efds[rxq_idx]; > + handle->elist[rxq_idx] =3D rev; > + ret =3D rte_epoll_ctl(epfd, EPOLL_CTL_ADD, rev.fd, > + &handle->elist[rxq_idx]); > + if (ret) { > + VHOST_LOG(ERR, "Add epoll event failed.\n"); > + return ret; > + } > + > + return 0; > +} > + > static int > eth_rxq_intr_enable(struct rte_eth_dev *dev, uint16_t qid) { > struct rte_vhost_vring vring; > struct vhost_queue *vq; > - int ret =3D 0; > + int old_intr_enable, ret =3D 0; >=20 > vq =3D dev->data->rx_queues[qid]; > if (!vq) { > @@ -537,6 +586,18 @@ eth_rxq_intr_enable(struct rte_eth_dev *dev, uint16_= t > qid) > return -1; > } >=20 > + rte_spinlock_lock(&vq->intr_lock); > + old_intr_enable =3D vq->intr_enable; > + vq->intr_enable =3D 1; > + ret =3D eth_vhost_update_intr(dev, qid); > + rte_spinlock_unlock(&vq->intr_lock); > + > + if (ret < 0) { > + VHOST_LOG(ERR, "Failed to update rxq%d's intr\n", qid); > + vq->intr_enable =3D old_intr_enable; > + return ret; > + } > + > ret =3D rte_vhost_get_vhost_vring(vq->vid, (qid << 1) + 1, &vring); > if (ret < 0) { > VHOST_LOG(ERR, "Failed to get rxq%d's vring\n", qid); @@ - > 571,6 +632,8 @@ eth_rxq_intr_disable(struct rte_eth_dev *dev, uint16_t qi= d) > rte_vhost_enable_guest_notification(vq->vid, (qid << 1) + 1, 0); > rte_wmb(); >=20 > + vq->intr_enable =3D 0; > + > return 0; > } >=20 > @@ -593,7 +656,6 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) { > struct rte_vhost_vring vring; > struct vhost_queue *vq; > - int count =3D 0; > int nb_rxq =3D dev->data->nb_rx_queues; > int i; > int ret; > @@ -623,6 +685,8 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) >=20 > VHOST_LOG(INFO, "Prepare intr vec\n"); > for (i =3D 0; i < nb_rxq; i++) { > + dev->intr_handle->intr_vec[i] =3D RTE_INTR_VEC_RXTX_OFFSET + > i; > + dev->intr_handle->efds[i] =3D -1; > vq =3D dev->data->rx_queues[i]; > if (!vq) { > VHOST_LOG(INFO, "rxq-%d not setup yet, skip!\n", i); > @@ -641,14 +705,12 @@ eth_vhost_install_intr(struct rte_eth_dev *dev) > "rxq-%d's kickfd is invalid, skip!\n", i); > continue; > } > - dev->intr_handle->intr_vec[i] =3D RTE_INTR_VEC_RXTX_OFFSET + > i; > dev->intr_handle->efds[i] =3D vring.kickfd; > - count++; > VHOST_LOG(INFO, "Installed intr vec for rxq-%d\n", i); > } >=20 > - dev->intr_handle->nb_efd =3D count; > - dev->intr_handle->max_intr =3D count + 1; > + dev->intr_handle->nb_efd =3D nb_rxq; > + dev->intr_handle->max_intr =3D nb_rxq + 1; > dev->intr_handle->type =3D RTE_INTR_HANDLE_VDEV; >=20 > return 0; > @@ -835,6 +897,7 @@ vring_conf_update(int vid, struct rte_eth_dev *eth_de= v, > uint16_t vring_id) { > struct rte_eth_conf *dev_conf =3D ð_dev->data->dev_conf; > struct pmd_internal *internal =3D eth_dev->data->dev_private; > + struct vhost_queue *vq; > struct rte_vhost_vring vring; > int rx_idx =3D vring_id % 2 ? (vring_id - 1) >> 1 : -1; > int ret =3D 0; > @@ -853,12 +916,18 @@ vring_conf_update(int vid, struct rte_eth_dev > *eth_dev, uint16_t vring_id) > vring_id); > return ret; > } > + eth_dev->intr_handle->efds[rx_idx] =3D vring.kickfd; >=20 > - if (vring.kickfd !=3D eth_dev->intr_handle->efds[rx_idx]) { > - VHOST_LOG(INFO, "kickfd for rxq-%d was changed.\n", > - rx_idx); > - eth_dev->intr_handle->efds[rx_idx] =3D vring.kickfd; > + vq =3D eth_dev->data->rx_queues[rx_idx]; > + if (!vq) { > + VHOST_LOG(ERR, "rxq%d is not setup yet\n", rx_idx); > + return -1; > } > + > + rte_spinlock_lock(&vq->intr_lock); > + if (vq->intr_enable) > + ret =3D eth_vhost_update_intr(eth_dev, rx_idx); > + rte_spinlock_unlock(&vq->intr_lock); > } >=20 > return ret; > @@ -1152,6 +1221,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, > uint16_t rx_queue_id, >=20 > vq->mb_pool =3D mb_pool; > vq->virtqueue_id =3D rx_queue_id * VIRTIO_QNUM + VIRTIO_TXQ; > + rte_spinlock_init(&vq->intr_lock); > dev->data->rx_queues[rx_queue_id] =3D vq; >=20 > return 0; > @@ -1173,6 +1243,7 @@ eth_tx_queue_setup(struct rte_eth_dev *dev, > uint16_t tx_queue_id, > } >=20 > vq->virtqueue_id =3D tx_queue_id * VIRTIO_QNUM + VIRTIO_RXQ; > + rte_spinlock_init(&vq->intr_lock); > dev->data->tx_queues[tx_queue_id] =3D vq; >=20 > return 0; > -- > 2.26.2 Reviewed-by: Chenbo Xia