From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 62E47A09EF; Mon, 21 Dec 2020 13:01:59 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 979C3CAAE; Mon, 21 Dec 2020 13:01:57 +0100 (CET) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by dpdk.org (Postfix) with ESMTP id 84288CAA9 for ; Mon, 21 Dec 2020 13:01:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1608552114; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+oDePNeMQelxkB1mAHbF8LZYaSyDOLDtF1bjz2IWUo0=; b=HN23DscX5YVK8Tu0yhb+9vw+Sdp8m6N/x9s+QKPMXClZzIqqKzgN5Y7iFWn6gGYLMiyGaK SqPOjHzJxzzsfHJJQx+5Uq7Lyw4xR9ACyNEoBHSTxpJvCp2v6NlSt0ZotWITYf4e2fNr+8 qJdZGz7JW4UmaqWDfes88HUlkjpEv/E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-582-s6gyharGNJCJ7vdBd6b9Ug-1; Mon, 21 Dec 2020 07:01:51 -0500 X-MC-Unique: s6gyharGNJCJ7vdBd6b9Ug-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 161D2C7405; Mon, 21 Dec 2020 12:01:50 +0000 (UTC) Received: from [10.36.110.46] (unknown [10.36.110.46]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8CCE760862; Mon, 21 Dec 2020 12:01:44 +0000 (UTC) From: Maxime Coquelin To: "Xia, Chenbo" , Thomas Monjalon , David Marchand Cc: dev , Stephen Hemminger , "Liang, Cunming" , "Lu, Xiuchun" , "Li, Miao" , "Wu, Jingjing" References: <20201218074736.93999-1-chenbo.xia@intel.com> <22c05dc3-bee9-4662-3f8d-a4dcd7635b42@redhat.com> Message-ID: Date: Mon, 21 Dec 2020 13:01:43 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <22c05dc3-bee9-4662-3f8d-a4dcd7635b42@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=maxime.coquelin@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] [PATCH 0/8] Introduce emudev library and iavf emudev driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 12/21/20 10:52 AM, Maxime Coquelin wrote: > Hi Chenbo, > > On 12/19/20 7:11 AM, Xia, Chenbo wrote: >> Hi David, >> >>> -----Original Message----- >>> From: David Marchand >>> Sent: Friday, December 18, 2020 5:54 PM >>> To: Xia, Chenbo >>> Cc: dev ; Thomas Monjalon ; Stephen >>> Hemminger ; Liang, Cunming >>> ; Lu, Xiuchun ; Li, Miao >>> ; Wu, Jingjing >>> Subject: Re: [PATCH 0/8] Introduce emudev library and iavf emudev driver >>> >>> On Fri, Dec 18, 2020 at 9:02 AM Chenbo Xia wrote: >>>> >>>> This series introduces a new device abstraction called emudev for >>> emulated >>>> devices. A new library (librte_emudev) is implemented. The first emudev >>>> driver is also introduced, which emulates Intel Adaptive Virtual >>> Function >>>> (iavf) as a software network device. >>>> >>>> This series has a dependency on librte_vfio_user patch series: >>>> http://patchwork.dpdk.org/cover/85389/ >>>> >>>> Background & Motivation >>>> ----------------------- >>>> The disaggregated/multi-process QEMU is using VFIO-over-socket/vfio-user >>>> as the main transport mechanism to disaggregate IO services from QEMU. >>>> Therefore, librte_vfio_user is introduced in DPDK to accommodate >>>> emulated devices for high performance I/O. Although vfio-user library >>>> provides possibility of emulating devices in DPDK, DPDK does not have >>>> a device abstraction for emulated devices. A good device abstraction >>> will >>>> be useful for applications or high performance data path driver. With >>>> this consideration, emudev library is designed and implemented. It also >>>> make it possbile to keep modular design on emulated devices by >>> implementing >>>> data path related logic in a standalone driver (e.g., an ethdev driver) >>>> and keeps the unrelated logic in the emudev driver. >>> >>> Since you mention performance, how does it compare to vhost-user/virtio? >> >> I think it depends on the device specification (i.e., how complex its data path >> handling is). A first try on iavf spec shows better performance than virtio >> in our simple tests. > > That's interesting! How big is the performance difference? And how do > we explain it? > > If there are improvements that could be done in the Virtio > specification, it would be great to know and work on their > implementations. It worries me a bit that every one is coming with > his new device emulation every release, making things like live- > migration difficult to achieve in the future. I did a quick review of the IAVF emudev driver to understand what other factors than ring layout could explain a performance gain. My understanding is that part of the performance gain may come from following things that are supported/implemented in Vhost-user backend and not in IAVF driver: 1. Memory hotplug. It seems the datapath is not safe against memory hotplug in the VM, which causes the memory tables to be updated asynchronously from the datapath. In order to support it in Vhost-user library, we had to introduce locks to ensure the datapath isn't accessing the shared memory while it is being remapped. 2. Live-migration. This feature does not seem supported in the driver, as I could not find dirty pages tracking mechanism. On Vhost-user side, supporting implies adding more branch conditions in the hot path, even when it is not used. 3. No IOMMU support. Same here, this is supported in Vhost-user library, and adding its support in IAVF driver would introduce some more branches in the hot path even when not used. 4. Out of bound memory accesses checks. While rte_iavf_emu_get_dma_vaddr() provides a way to ensure the full requested length is mapped, the data path does not use it. It does not even ensure the translated address is non-NULL. It makes it trivial for a malicious guest to make the hypervisor's vSwitch to crash by simply passing random values as buffer's address and length. Fixing it is trivial, but it will add several more checks and loops (if a buffer is spanned across two pages) in the hot path. Other than that, there is for sure a performance gain due to all the features Virtio-net supports that we have to check and handle in the hotpath, like indirect descriptors or mergeable buffers for example. Best regards, Maxime > Regards, > Maxime > >> Thanks! >> Chenbo >> >>> >>> >>> -- >>> David Marchand >>