From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f170.google.com (mail-pf0-f170.google.com [209.85.192.170]) by dpdk.org (Postfix) with ESMTP id 37F608E8C for ; Mon, 28 Dec 2015 12:06:42 +0100 (CET) Received: by mail-pf0-f170.google.com with SMTP id 78so110958799pfw.2 for ; Mon, 28 Dec 2015 03:06:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=igel-co-jp.20150623.gappssmtp.com; s=20150623; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=GqXeezbjDiF7Io31QX1QWMcj4+cB6lCIOhE+5HW+R5I=; b=bigmhxOTcol3hn53daE1i4xU7GRlsa4CaRQeFPRlb8shPb/u4vfxfPGMQ3Q2Ev7EDh Wl5U2kGOnEqBtrzEM6apmlRE8NPOwbJhOUFnfIRkLDD8Zh3CdQ+CJCxtux8bAkw8j1pw ndFFZyYB0blK7NeeMoBVnxOhmGoRTF9jJ2qF6zJwem24buBxcWdliNC/+gEmhGxbaGuA T0eDYg/IRTweUkS0+G59ERZKoaB0M/nQTMO2jpGYN0hR23tUjckPAVR2S4z1Cm9OPZSx Ty0iviYkN1nHNSuDAOd81jBp5i6k1hpn0ZWTJ4MX8yOMfRJr68x1z+KKY+ohNSLhgZpQ BgvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=GqXeezbjDiF7Io31QX1QWMcj4+cB6lCIOhE+5HW+R5I=; b=GNOD71zZtrpnKBdoKjYTzjac7v0Y1MtzCAlBeltKd4ZWEj1UYsWCTgfRoIX19sEw1V Jk0HuetbwNWD+g79o/DjdwFS+Bks1psyApg4tnCoRdrjilLlJDGMOGVtTOF6JNajFKqg NIAPRfWvoV28o/ifGhD5V38ouZVUI3Gy5qw5QZRmo43zu5ETr/nwbHoF3AorDj3lUKvh S+88wbftwDOlqRZz3h84zG755eNCUZ8lSrRPtLsM497lsRhHxtDYODe/edT5ZieqCZ2f seSs1FqXhpsaA1cmi5Vm+veElYO5gS3d+3W8qxgMLa8Hzz6ruyCkzV14bHMhtHC44PTs yS6g== X-Gm-Message-State: ALoCoQlBrHIe+6bN2Eh6YDlOxdw88ihECct0eEPG0Gp9Op9YfGkSOQpS4DG2B947SW2rPi/sbT8aFWZQiOfwlIW0m1w/xN2jrQ== X-Received: by 10.98.14.9 with SMTP id w9mr61428000pfi.146.1451300801616; Mon, 28 Dec 2015 03:06:41 -0800 (PST) Received: from [192.168.1.11] (p2734229-ipngn22401marunouchi.tokyo.ocn.ne.jp. [153.209.71.229]) by smtp.googlemail.com with ESMTPSA id l66sm75868714pfj.71.2015.12.28.03.06.38 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 Dec 2015 03:06:41 -0800 (PST) To: "Qiu, Michael" , "dev@dpdk.org" References: <1447930650-26023-1-git-send-email-mukawa@igel.co.jp> <533710CFB86FA344BFBF2D6802E6028622F04DCF@SHSMSX101.ccr.corp.intel.com> From: Tetsuya Mukawa X-Enigmail-Draft-Status: N1110 Message-ID: <568117BD.1040301@igel.co.jp> Date: Mon, 28 Dec 2015 20:06:37 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <533710CFB86FA344BFBF2D6802E6028622F04DCF@SHSMSX101.ccr.corp.intel.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: "nakajima.yoshihiro@lab.ntt.co.jp" , "zhbzg@huawei.com" , "mst@redhat.com" , "gaoxiaoqiu@huawei.com" , "oscar.zhangbo@huawei.com" , "ann.zhuangyanying@huawei.com" , "zhoujingbin@huawei.com" , "guohongzhen@huawei.com" Subject: Re: [dpdk-dev] [RFC PATCH 0/2] Virtio-net PMD Extension to work on host. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Dec 2015 11:06:42 -0000 On 2015/12/28 14:15, Qiu, Michael wrote: > Hi, Tetsuya > > I have a question about your solution, as I know you plan to run qemu > and dpdk both in container right? Hi Michael, Thanks for your comments. It depends on use cases. For example, if we want to use vhost-user, we need to invoke 3 processes (DPDK app that uses virtio-net PMD, QEMU and DPDK app that uses vhost-user PMD). These 3 process can run in both container and host. Some users may want to run 3 processes in different container. Some users may want to run QEMU and vhost-user PMD prcess in one container. Anyway, in some cases, DPDK app and QEMU will run in one container like you said. > > If so, I think it's a bit tricky, DPDK is a lib, and qemu is a App, > seems it is not suitable to let a lib depends on Apps. If we implement as Jianfeng did, virtio-net PMD will depends on vhost-user PMD process. And if there is no vhost-user process, it doesn't work. In my case, if there is no QEMU process, it doesn't work. > Also, till now I don't see any usecase to run qemu inside container. Described above, if you don't want to run QEMU in container, you don't need to do. You can run QEMU in host. Actually, QEMU will handle file descriptor of virtio-net PMD process memory, so someone may want to isolate it. In this case, they can run QEMU in container. Thanks, Tetsuya > Thanks, > Michael > > On 11/19/2015 6:58 PM, Tetsuya Mukawa wrote: >> THIS IS A PoC IMPLEMENATION. >> >> [Abstraction] >> >> Normally, virtio-net PMD only works on VM, because there is no virtio-net device on host. >> This RFC patch extends virtio-net PMD to be able to work on host as virtual PMD. >> But we didn't implement virtio-net device as a part of virtio-net PMD. >> To prepare virtio-net device for the PMD, start QEMU process with special QTest mode, then connect it from virtio-net PMD through unix domain socket. >> >> The PMD can connect to anywhere QEMU virtio-net device can. >> For example, the PMD can connects to vhost-net kernel module and vhost-user backend application. >> Similar to virtio-net PMD on QEMU, application memory that uses virtio-net PMD will be shared between vhost backend application. >> But vhost backend application memory will not be shared. >> >> Main target of this PMD is container like docker, rkt, lxc and etc. >> We can isolate related processes(virtio-net PMD process, QEMU and vhost-user backend process) by container. >> But, to communicate through unix domain socket, shared directory will be needed. >> >> >> [How to use] >> >> So far, we need QEMU patch to connect to vhost-user backend. >> Please check known issue in later section. >> Because of this, I will describe example of using vhost-net kernel module. >> >> - Compile >> Set "CONFIG_RTE_VIRTIO_VDEV=y" in config/common_linux. >> Then compile it. >> >> - Start QEMU like below. >> $ sudo qemu-system-x86_64 -qtest unix:/tmp/qtest0,server -machine accel=qtest \ >> -display none -qtest-log /dev/null \ >> -netdev type=tap,script=/etc/qemu-ifup,id=net0,vhost=on \ >> -device virtio-net-pci,netdev=net0 \ >> -chardev socket,id=chr1,path=/tmp/ivshmem0,server \ >> -device ivshmem,size=1G,chardev=chr1,vectors=1 >> >> - Start DPDK application like below >> $ sudo ./testpmd -c f -n 1 -m 1024 --shm \ >> --vdev="eth_cvio0,qtest=/tmp/qtest0,ivshmem=/tmp/ivshmem0" -- \ >> --disable-hw-vlan --txqflags=0xf00 -i >> >> - Check created tap device. >> >> (*1) Please Specify same memory size in QEMU and DPDK command line. >> >> >> [Detailed Description] >> >> - virtio-net device implementation >> The PMD uses QEMU virtio-net device. To do that, QEMU QTest functionality is used. >> QTest is a test framework of QEMU devices. It allows us to implement a device driver outside of QEMU. >> With QTest, we can implement DPDK application and virtio-net PMD as standalone process on host. >> When QEMU is invoked as QTest mode, any guest code will not run. >> To know more about QTest, see below. >> http://wiki.qemu.org/Features/QTest >> >> - probing devices >> QTest provides a unix domain socket. Through this socket, driver process can access to I/O port and memory of QEMU virtual machine. >> The PMD will send I/O port accesses to probe pci devices. >> If we can find virtio-net and ivshmem device, initialize the devices. >> Also, I/O port accesses of virtio-net PMD will be sent through socket, and virtio-net PMD can initialize vitio-net device on QEMU correctly. >> >> - ivshmem device to share memory >> To share memory that virtio-net PMD process uses, ivshmem device will be used. >> Because ivshmem device can only handle one file descriptor, shared memory should be consist of one file. >> To allocate such a memory, EAL has new option called "--shm". >> If the option is specified, EAL will open a file and allocate memory from hugepages. >> While initializing ivshmem device, we can set BAR(Base Address Register). >> It represents which memory QEMU vcpu can access to this shared memory. >> We will specify host physical address of shared memory as this address. >> It is very useful because we don't need to apply patch to QEMU to calculate address offset. >> (For example, if virtio-net PMD process will allocate memory from shared memory, then specify the physical address of it to virtio-net register, QEMU virtio-net device can understand it without calculating address offset.) >> >> - Known limitation >> So far, the PMD doesn't handle interrupts from QEMU devices. >> Because of this, VIRTIO_NET_F_STATUS functionality is dropped. >> But without it, we can use all virtio-net functions. >> >> - Known issues >> So far, to use vhost-user, we need to apply vhost-user patch to QEMU and DPDK vhost library. >> This is because, QEMU will not send memory information and file descriptor of ivshmem device to vhost-user backend. >> (Anyway, vhost-net kernel module can receive the information. So vhost-user behavior will not be correct. I will submit the patch to QEMU soon) >> Also, we may have an issue in DPDK vhost library to handle kickfd and callfd. The patch for it is needed. >> (Let me check it more) >> If someone wants to check vhost-user behavior, I will describe it more in later email. >> >> >> [Addition] >> >> We can apply same manner to handle any kind of QEMU devices from DPDK application. >> So far, I don't have any ideas except for virtio-net device. But someone would have. >> >> >> Tetsuya Mukawa (2): >> EAL: Add new EAL "--shm" option. >> virtio: Extend virtio-net PMD to support container environment >> >> config/common_linuxapp | 5 + >> drivers/net/virtio/Makefile | 4 + >> drivers/net/virtio/qtest.c | 590 +++++++++++++++++++++++++++++ >> drivers/net/virtio/virtio_ethdev.c | 214 ++++++++++- >> drivers/net/virtio/virtio_ethdev.h | 16 + >> drivers/net/virtio/virtio_pci.h | 25 ++ >> lib/librte_eal/common/eal_common_options.c | 5 + >> lib/librte_eal/common/eal_internal_cfg.h | 1 + >> lib/librte_eal/common/eal_options.h | 2 + >> lib/librte_eal/common/include/rte_memory.h | 5 + >> lib/librte_eal/linuxapp/eal/eal_memory.c | 71 ++++ >> 11 files changed, 917 insertions(+), 21 deletions(-) >> create mode 100644 drivers/net/virtio/qtest.c >>