From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 2F891293C for ; Fri, 4 Mar 2016 07:10:22 +0100 (CET) Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga101.fm.intel.com with ESMTP; 03 Mar 2016 22:10:21 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,534,1449561600"; d="scan'208";a="663629822" Received: from shwdeisgchi083.ccr.corp.intel.com (HELO [10.239.67.193]) ([10.239.67.193]) by FMSMGA003.fm.intel.com with ESMTP; 03 Mar 2016 22:10:18 -0800 To: Tetsuya Mukawa , dev@dpdk.org References: <1455075613-3605-3-git-send-email-mukawa@igel.co.jp> <1456129075-14909-6-git-send-email-mukawa@igel.co.jp> <56D8F06F.3060705@intel.com> <56D91792.1060406@igel.co.jp> From: "Tan, Jianfeng" Message-ID: <56D926CA.80206@intel.com> Date: Fri, 4 Mar 2016 14:10:18 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56D91792.1060406@igel.co.jp> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v3 5/6] virtio: Add support for qtest virtio-net PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2016 06:10:22 -0000 Hi Tetsuya, On 3/4/2016 1:05 PM, Tetsuya Mukawa wrote: > On 2016/03/04 11:18, Tan, Jianfeng wrote: >> Hi Tetsuya, >> >> Seems that this patch is too long. Is it possible to split into >> multiple commits? > Hi Jianfeng, > > Sure, will do. > >> On 2/22/2016 4:17 PM, Tetsuya Mukawa wrote: >>> The patch adds a new virtio-net PMD configuration that allows the PMD to >>> work on host as if the PMD is in VM. >>> Here is new configuration for virtio-net PMD. >>> - CONFIG_RTE_VIRTIO_VDEV_QTEST >>> To use this mode, EAL needs map all hugepages as one file. Also the file >>> should be mapped between (1 << 31) and (1 << 44). And start address >>> should be aligned by EAL memory size. >>> >>> To allocate like above, use below options. >>> --single-file >>> --range-virtaddr=0x80000000-0x100000000000 >>> --align-memsize >>> If a free regions isn't found, EAL will return error. >>> >>> To prepare virtio-net device on host, the users need to invoke QEMU >>> process in special qtest mode. This mode is mainly used for testing QEMU >>> devices from outer process. In this mode, no guest runs. >>> Here is QEMU command line. >>> >>> $ qemu-system-x86_64 \ >>> -machine pc-i440fx-1.4,accel=qtest \ >>> -display none -qtest-log /dev/null \ >>> -qtest unix:/tmp/socket,server \ >>> -netdev type=tap,script=/etc/qemu-ifup,id=net0,queues=1 \ >>> -device >>> virtio-net-pci,netdev=net0,mq=on,disable-modern=false,addr=3 \ >>> -chardev socket,id=chr1,path=/tmp/ivshmem,server \ >>> -device ivshmem,size=1G,chardev=chr1,vectors=1,addr=4 >>> >>> * Should use qemu-2.5.1, or above. >>> * QEMU process is needed per port. >>> * virtio-1.0 device are only supported. >>> * The vhost backends like vhost-net and vhost-user can be specified. >>> * In most cases, just using above command is enough, but you can also >>> specify other QEMU virtio-net options. >>> * Only checked "pc-i440fx-1.4" machine, but may work with other >>> machines. >>> * Should not add "--enable-kvm" to QEMU command line. >> Correct me if wrong: all control msgs go through qemu process, e.g., >> tx notifications and rx interrupts need follow frontend-qemu-backend >> path. Question: qemu is started without --enable-kvm, as I understand, >> ioeventfd, the basis of kickfd/callfd, will not be available. So how >> does qemu kick backend or be kicked by backend? > Actually, vhost-backend process will receive kickfd and callfd as -1. > (Currently, we have a bug in librte_vhost, because the library treats -1 > as "not initialized state". But actually without "--enable-kvm", -1 will > be set by qemu to initialize kickfd and callfd. I will send a patch for > the issue with next patch series.) Yes, we noticed the problem too: librte_vhost judges virtio_is_ready by whether both fds are set. But except that, what's kernel's way to do the judgement? In addition, it would be better to be a independent fix patch. > > In our case, virtio-net driver and vhost-backend driver are PMD. So we > don't use kickfd and callfd, right? > > If you worried about vhost-net case, vhost-net kernel thread will work > without ioeventfd and irqfd. > In this case, virtio-net PMD can kick the vhost-net by accessing > VIRTIO_PCI_QUEUE_NOTIFY register. > (vhost-net doesn't need to kick virtio-net driver, because the driver is > PMD.) I ask this question because I think interrupt mode will help the scalability. Return to your solution, by accessing VIRTIO_PCI_QUEUE_NOTIFY register, virtio-net PMD can only wake up qemu, but how does qemu wakes up vhost-net under the case that kickfd = callfd = -1. > >>> After invoking QEMU, the PMD can connect to QEMU process using unix >>> domain sockets. Over these sockets, virtio-net, ivshmem and piix3 >>> device in QEMU are probed by the PMD. >>> Here is example of command line. >>> >>> $ testpmd -c f -n 1 -m 1024 --no-pci --single-file \ >>> --range-virtaddr=0x80000000-0x100000000000 --align-memsize \ >>> >>> --vdev="eth_qtest_virtio0,qtest=/tmp/socket,ivshmem=/tmp/ivshmem"\ >>> -- --disable-hw-vlan --txqflags=0xf00 -i >>> >>> Please specify same unix domain sockets and memory size in both QEMU >>> and DPDK command lines like above. >>> The share memory size should be power of 2, because ivshmem only >>> accepts such memory size. >>> >>> Signed-off-by: Tetsuya Mukawa >>> --- >>> config/common_linuxapp | 1 + >>> drivers/net/virtio/Makefile | 4 + >>> drivers/net/virtio/qtest.c | 1342 >>> ++++++++++++++++++++++++++++++++++++ >>> drivers/net/virtio/qtest.h | 65 ++ >>> drivers/net/virtio/virtio_ethdev.c | 383 +++++++++- >>> drivers/net/virtio/virtio_pci.c | 364 +++++++++- >>> drivers/net/virtio/virtio_pci.h | 5 +- >>> 7 files changed, 2122 insertions(+), 42 deletions(-) >>> create mode 100644 drivers/net/virtio/qtest.c >>> create mode 100644 drivers/net/virtio/qtest.h >>> >>> diff --git a/config/common_linuxapp b/config/common_linuxapp >>> index 452f39c..f6e53bc 100644 >>> --- a/config/common_linuxapp >>> +++ b/config/common_linuxapp >>> @@ -533,3 +533,4 @@ CONFIG_RTE_TEST_PMD_RECORD_BURST_STATS=n >>> # Enable virtio support for container >>> # >>> CONFIG_RTE_VIRTIO_VDEV=y >>> +CONFIG_RTE_VIRTIO_VDEV_QTEST=y >>> diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile >>> index ef920f9..6c11378 100644 >>> --- a/drivers/net/virtio/Makefile >>> +++ b/drivers/net/virtio/Makefile >>> @@ -56,6 +56,10 @@ ifeq ($(CONFIG_RTE_VIRTIO_VDEV),y) >>> SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += vhost_embedded.c >>> endif >>> +ifeq ($(CONFIG_RTE_VIRTIO_VDEV_QTEST),y) >>> + SRCS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += qtest.c >>> +endif >>> + >>> # this lib depends upon: >>> DEPDIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += lib/librte_eal >>> lib/librte_ether >>> DEPDIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += lib/librte_mempool >>> lib/librte_mbuf >>> diff --git a/drivers/net/virtio/qtest.c b/drivers/net/virtio/qtest.c >>> new file mode 100644 >>> index 0000000..061aab5 >>> --- /dev/null >>> +++ b/drivers/net/virtio/qtest.c >>> @@ -0,0 +1,1342 @@ >>> +/*- >>> + * BSD LICENSE >>> + * >>> + * Copyright(c) 2016 IGEL Co., Ltd. All rights reserved. >>> + * All rights reserved. >>> + * >>> + * Redistribution and use in source and binary forms, with or without >>> + * modification, are permitted provided that the following conditions >>> + * are met: >>> + * >>> + * * Redistributions of source code must retain the above copyright >>> + * notice, this list of conditions and the following disclaimer. >>> + * * Redistributions in binary form must reproduce the above >>> copyright >>> + * notice, this list of conditions and the following >>> disclaimer in >>> + * the documentation and/or other materials provided with the >>> + * distribution. >>> + * * Neither the name of IGEL Co., Ltd. nor the names of its >>> + * contributors may be used to endorse or promote products >>> derived >>> + * from this software without specific prior written permission. >>> + * >>> + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND >>> CONTRIBUTORS >>> + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT >>> + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND >>> FITNESS FOR >>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE >>> COPYRIGHT >>> + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, >>> INCIDENTAL, >>> + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT >>> + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS >>> OF USE, >>> + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND >>> ON ANY >>> + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR >>> TORT >>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF >>> THE USE >>> + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH >>> DAMAGE. >>> + */ >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> +#include >>> + >>> +#include >>> +#include >>> +#include >>> +#include >>> + >>> +#include "virtio_pci.h" >>> +#include "virtio_logs.h" >>> +#include "virtio_ethdev.h" >>> +#include "qtest.h" >>> + >>> +#define NB_BAR 6 >>> + >>> +/* PIIX3 configuration registers */ >>> +#define PIIX3_REG_ADDR_PIRQA 0x60 >>> +#define PIIX3_REG_ADDR_PIRQB 0x61 >>> +#define PIIX3_REG_ADDR_PIRQC 0x62 >>> +#define PIIX3_REG_ADDR_PIRQD 0x63 >>> + >>> +/* Device information */ >>> +#define VIRTIO_NET_DEVICE_ID 0x1000 >>> +#define VIRTIO_NET_VENDOR_ID 0x1af4 >>> +#define VIRTIO_NET_IRQ_NUM 10 >>> +#define IVSHMEM_DEVICE_ID 0x1110 >>> +#define IVSHMEM_VENDOR_ID 0x1af4 >>> +#define IVSHMEM_PROTOCOL_VERSION 0 >>> +#define PIIX3_DEVICE_ID 0x7000 >>> +#define PIIX3_VENDOR_ID 0x8086 >>> + >>> +/* ------------------------------------------------------------ >>> + * IO port mapping of qtest guest >>> + * ------------------------------------------------------------ >>> + * 0x0000 - 0xbfff : not used >>> + * 0xc000 - 0xc03f : virtio-net(BAR0) >>> + * 0xc040 - 0xffff : not used >>> + * >>> + * ------------------------------------------------------------ >>> + * Memory mapping of qtest quest >>> + * ------------------------------------------------------------ >>> + * 0x00000000_00000000 - 0x00000000_3fffffff : not used >>> + * 0x00000000_40000000 - 0x00000000_40000fff : virtio-net(BAR1) >>> + * 0x00000000_40001000 - 0x00000000_40ffffff : not used >>> + * 0x00000000_41000000 - 0x00000000_417fffff : virtio-net(BAR4) >>> + * 0x00000000_41800000 - 0x00000000_41ffffff : not used >>> + * 0x00000000_42000000 - 0x00000000_420000ff : ivshmem(BAR0) >>> + * 0x00000000_42000100 - 0x00000000_42ffffff : not used >>> + * 0x00000000_80000000 - 0xffffffff_ffffffff : ivshmem(BAR2) >>> + * >> Is it possible to arrange multiple virtio-net devices here? What's the >> challenges? > Yes, you can manage multiple virtio-net devices here, if you define > correct memory map. So this memory map will be decided here or in qemu? > >> Seems that lots of below code do the same work as libqos. So can we >> just link libqos? Or we need to maintain this code. > Problem is libqos will be GPL. > So I wrote the code from scratch. OK, great! To make it extensible for other virtio devices (like scsi), we need to abstract those code as a qtest utils, and virtio-net related code into other .c file.