From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EAA86A0547; Mon, 30 Aug 2021 20:10:53 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 73D6640DF7; Mon, 30 Aug 2021 20:10:53 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mails.dpdk.org (Postfix) with ESMTP id 0963340141 for ; Mon, 30 Aug 2021 20:10:51 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630347051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v7U7BwSrh+MN8d1GtHNgcEgJTSIa0YvQifRGUsvyrn0=; b=QyXrYySmgbL85LdyCEKwzxyLfx3QZAGl04YqCO33OyvmuiFzdjplU2Wd/prKvebBRhOkg2 SDncKfzrDxaheT8DtVP390FMwUYjTeRhhp2n1UK7GhoHkBUmvmOQmqSMKYP5zcP8dRtiux pU8Ry+hEqIUYztwEsLTeod02KhfMAi4= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-300-ZMVzz1QJPuOta8YGnZojtQ-1; Mon, 30 Aug 2021 14:10:49 -0400 X-MC-Unique: ZMVzz1QJPuOta8YGnZojtQ-1 Received: by mail-qk1-f198.google.com with SMTP id o4-20020ae9f504000000b003d39d97b227so245186qkg.2 for ; Mon, 30 Aug 2021 11:10:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=v7U7BwSrh+MN8d1GtHNgcEgJTSIa0YvQifRGUsvyrn0=; b=MW7Fo/+c3y0einEzrgKyAJWdhFILACqqzQ5/JIjpDBRYMxl2QHK5bI2svh0gioxUrS Gvwr/ycuAXSbVcE1dpzyPnNi780PxRx1qDvYUZTL0ajgqo/+kiw+LOcjwdb9QCF6Cols ZUA1jnfdjF7vfHhjns6nfM7R3Z9j6/KM8SxmvhQETaq+oAxOijflH4WLBo1EDnGWLDgG 2/3O9/CWVGhQNChBqm4RvSgnQaxuCQyEpSIFNf/hPhRF9sWK6JIfjXZSh+YJUTjMJs/D VDg+zWEVtGaEXaz4rKCKgS4Ff6V9AntIeeDPDqyoh2sucKC2FE7XQQuUUHRsGRmXlAaD UMNg== X-Gm-Message-State: AOAM531PYeWtpRoyaKrVCv8kOW5hb687BbSe1wBjQEAWpj449nBePp6u UAmKcDJjjmWboCkOagh+Wpa7oEZTRJ0z4udrmVN2pVZLsmOnHDWHvy+r2DpO8D22u09a1hY4Pqb g8WdzK9t6VWEZXhIZG+0= X-Received: by 2002:ac8:4e2b:: with SMTP id d11mr22386956qtw.384.1630347048854; Mon, 30 Aug 2021 11:10:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz2sWdh2Sy+x/qjIUCrdI3S8mN/dnMxI9llDxtnrf8lqSllhc6MOT7RUddR/kOpVfDUJ1LUvW04TiHoIi6Mf3M= X-Received: by 2002:ac8:4e2b:: with SMTP id d11mr22386937qtw.384.1630347048665; Mon, 30 Aug 2021 11:10:48 -0700 (PDT) MIME-Version: 1.0 References: <20210827161231.579968-1-eperezma@redhat.com> In-Reply-To: From: Eugenio Perez Martin Date: Mon, 30 Aug 2021 20:10:12 +0200 Message-ID: To: "Xia, Chenbo" Cc: Maxime Coquelin , "dev@dpdk.org" , Pei Zhang , Jason Wang Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH] vhost: Clean iotlb cache on vring stop X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Aug 30, 2021 at 1:58 PM Xia, Chenbo wrote: > > Hi Eugenio, > Hi Chenbo, > > -----Original Message----- > > From: Eugenio P=C3=A9rez > > Sent: Saturday, August 28, 2021 12:13 AM > > To: Maxime Coquelin ; Xia, Chenbo > > > > Cc: dev@dpdk.org; Pei Zhang ; Jason Wang > > > > Subject: [PATCH] vhost: Clean iotlb cache on vring stop > > Clean -> clean > Is that something I need to send a new revision for, or is it ok to apply on the maintainer side? > > > > Old IOVA cache entries are left when there is a change on virtio driver > > in VM. In case that all these old entries have iova addresses lesser > > than new iova entries, vhost code will need to iterate all the cache to > > find the new ones. In case of just a new iova entry needed for the new > > translations, this condition will last forever. > > > > This has been observed in virtio-net to testpmd's vfio-pci driver > > transition, reducing the performance from more than 10Mpps to less than > > 0.07Mpps if the hugepage address was higher than the networking > > buffers. Since all new buffers are contained in this new gigantic page, > > vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst. > > I'm curious why QEMU will not invalidate iotlb when virtio-net driver is = removed > (dma region should be unmapped). > I'm going to investigate this more, but qemu iommu notifier callback (vhost_iommu_unmap_notify) is never called through all the test. Also, guest kernel code calls dma_unmap_page for each buffer and vqs, but it never generates an iotlb flush. Or do you mean that qemu should also flush all iotlb entries on vhost device stop? > And since the perf drop is huge, why not cc to stable and add fix tag? > I was not sure if it was worth it to backport, but I would say that the issue can be reproduced with enough bad luck. Since translations have always been saved in a linked list: Fixes: d012d1f293f4 ("vhost: add IOTLB helper functions") Same question as before, if no changes to the code are needed for the patch, do I need to send a second revision? Thanks! > Thanks, > Chenbo > > > > > Signed-off-by: Eugenio P=C3=A9rez > > Reported-by: Pei Zhang > > --- > > lib/vhost/vhost_user.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/lib/vhost/vhost_user.c b/lib/vhost/vhost_user.c > > index 29a4c9af60..7de48f5333 100644 > > --- a/lib/vhost/vhost_user.c > > +++ b/lib/vhost/vhost_user.c > > @@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pde= v, > > msg->size =3D sizeof(msg->payload.state); > > msg->fd_num =3D 0; > > > > + vhost_user_iotlb_flush_all(vq); > > + > > vring_invalidate(dev, vq); > > > > return RTE_VHOST_MSG_RESULT_REPLY; > > -- > > 2.27.0 >