From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by dpdk.org (Postfix) with ESMTP id D8A281B7DB for ; Thu, 8 Feb 2018 10:08:59 +0100 (CET) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 665E776F94; Thu, 8 Feb 2018 09:08:59 +0000 (UTC) Received: from [10.36.112.24] (ovpn-112-24.ams2.redhat.com [10.36.112.24]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9DF452026987; Thu, 8 Feb 2018 09:08:57 +0000 (UTC) To: "Wang, Xiao W" , "dev@dpdk.org" Cc: "Tan, Jianfeng" , "Bie, Tiwei" , "yliu@fridaylinux.org" , "Liang, Cunming" , "Daly, Dan" , "Wang, Zhihong" References: <20180204145542.38345-1-xiao.w.wang@intel.com> <20180204145542.38345-3-xiao.w.wang@intel.com> From: Maxime Coquelin Message-ID: <1909c444-4b88-bbb3-9fc2-b85ac8a3f5cb@redhat.com> Date: Thu, 8 Feb 2018 10:08:55 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 08 Feb 2018 09:08:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Thu, 08 Feb 2018 09:08:59 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'maxime.coquelin@redhat.com' RCPT:'' Subject: Re: [dpdk-dev] [PATCH 2/3] net/vdpa_virtio_pci: introduce vdpa sample driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Feb 2018 09:09:00 -0000 Hi Xiao, On 02/08/2018 03:23 AM, Wang, Xiao W wrote: > Hi Maxime, > >> -----Original Message----- >> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com] >> Sent: Tuesday, February 6, 2018 10:24 PM >> To: Wang, Xiao W ; dev@dpdk.org >> Cc: Tan, Jianfeng ; Bie, Tiwei ; >> yliu@fridaylinux.org; Liang, Cunming ; Daly, Dan >> ; Wang, Zhihong >> Subject: Re: [PATCH 2/3] net/vdpa_virtio_pci: introduce vdpa sample driver >> >> Hi Xiao, >> >> On 02/04/2018 03:55 PM, Xiao Wang wrote: >>> This driver is a reference sample of making vDPA device driver based >>> on vhost lib, this driver uses a standard virtio-net PCI device as >>> vDPA device, it can serve as a backend for a virtio-net pci device >>> in nested VM. >>> >>> The key driver ops implemented are: >>> >>> * vdpa_virtio_eng_init >>> Mapping virtio pci device with VFIO into userspace, and read device >>> capability and intialize internal data. >>> >>> * vdpa_virtio_eng_uninit >>> Release the mapped device. >>> >>> * vdpa_virtio_info_query >>> Device capability reporting, e.g. queue number, features. >>> >>> * vdpa_virtio_dev_config >>> With the guest virtio information provideed by vhost lib, this >>> function configures device and IOMMU to set up vhost datapath, >>> which includes: Rx/Tx vring, VFIO interrupt, kick relay. >>> >>> * vdpa_virtio_dev_close >>> Unset the stuff that are configured previously by dev_conf. >>> >>> This driver requires the virtio device supports VIRTIO_F_IOMMU_PLATFORM >>> , because the buffer address written in desc is IOVA. >>> >>> Because vDPA driver needs to set up MSI-X vector to interrupt the guest, >>> only vfio-pci is supported currently. >>> >>> Signed-off-by: Xiao Wang >>> --- >>> config/common_base | 6 + >>> config/common_linuxapp | 1 + >>> drivers/net/Makefile | 1 + >>> drivers/net/vdpa_virtio_pci/Makefile | 31 + >>> .../net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c | 1527 >> ++++++++++++++++++++ >>> .../rte_vdpa_virtio_pci_version.map | 4 + >>> mk/rte.app.mk | 1 + >>> 7 files changed, 1571 insertions(+) >>> create mode 100644 drivers/net/vdpa_virtio_pci/Makefile >>> create mode 100644 drivers/net/vdpa_virtio_pci/rte_eth_vdpa_virtio_pci.c >>> create mode 100644 >> drivers/net/vdpa_virtio_pci/rte_vdpa_virtio_pci_version.map >> >> Is there a specific constraint that makes you expose PCI functions and >> duplicate a lot of vfio code into the driver? > > The existing vfio code doesn't fit VDPA well, this vDPA driver needs to program IOMMU for a vDPA device with a VM's memory table. > While the eal/vfio uses a struct vfio_cfg to takes all regular devices and add them to a single vfio_container, and program IOMMU with DPDK process's memory table. > > This driver doing PCI VFIO initialization itself can avoid affecting the global vfio_cfg structure. Ok, I get it. So I think what you have to do is to extend eal/vfio for this case. Or at least, have a vdpa layer to perform this, else every offload driver will have to duplicate the code. >> >> Wouldn't it be better (if possible) to use RTE_PMD_REGISTER_PCI() & co. >> to benefit from all the existing infrastructure? > > RTE_PMD_REGISTER_PCI() & co will make this driver as PCI driver (physical device), then this will conflict with the virtio_pmd. > So I make vDPA device driver as a vdev driver. Yes, but it is a PCI device, not a virtual device. You have to extend the EAL to support this new class of devices/drivers. Think of it as in kernel when a NIC device can be either binded to its NIC driver, VFIO or UIO. If I look at patch 3, you have to set --no-pci, or at least I think to blacklist the Virtio device. I wonder if real vDPA cards will support either vDPA mode or or behave like a regular NIC, like the Virtio case in your example. If this is the case, maybe the vDPA code for a NIC could be in the same driver as the "NIC" mode. A new struct rte_pci_driver driver flag could be introduced to specify that the driver supports vDPA. Then, in EAL arguments, if a vhost vdev specifies it wants Virtio device at PCI addr 00:01:00 as offload, the PCI layer could probe this device in "vdpa" mode. Also, I don't know if this will be possible with real vDPA cards, but we could have the application doing packet switching between vhost-user vdev and the Virtio device. And at some point, at runtime, switch into vDPA mode. This use-case would be much easier to implement if vDPA relied on existing PCI layer. I may be not very clear, don't hesitate to ask questions. But generally, I think vDPA has to fit in existing DPDK architecture, and not try to live outside of it. Thanks, Maxime >> >> Maxime > > Thanks for the comments, > Xiao >