From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B232DA04AF; Fri, 14 Aug 2020 17:00:11 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3BEF11C0C3; Fri, 14 Aug 2020 17:00:10 +0200 (CEST) Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by dpdk.org (Postfix) with ESMTP id 1F61F1BFF3 for ; Fri, 14 Aug 2020 17:00:09 +0200 (CEST) Received: by mail-pl1-f196.google.com with SMTP id k13so4282106plk.13 for ; Fri, 14 Aug 2020 08:00:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=na9qURyqNa0nTzILMkhoxEgDFWeyphbQDPYl2d5CycE=; b=i38QYnFZkhezZ7Y95fNtUyK8GMKTStQHoBszF/4mVfJFaZdrvSemgSi1X3zxpmtN9c i294sIzytOwbG2YNWeVoNKsOBrqT/w/R4NpuaTtkZKeizdUxHKPMkfCWK2Z7/JOQ8qOm 3J8I7pMIQt7z5rWHMbz/JJUdTIG0Mxzfp4pVmWGNMqRFQ6rt+BdhaVBybj3YyymVg+5i T92PY21+Ccn4r/QEAAnX2/gpIK6KxKSWaO8M79MtQgeGCJsU7drzcO1iCD5OKobOCEa9 R5LLSSRi3q1TF2iTpb7TN0BtUkILi/dFbYky4yS6uAELrf/Sdjckt3Q00FTwFtilTQoV bW6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=na9qURyqNa0nTzILMkhoxEgDFWeyphbQDPYl2d5CycE=; b=NgYzOAwhdTK0U69R6Xy+vDR9WaOiDcbOMOIRJTzI+ddOMchizO9oPJlfAla3kUXIAb 8q2PyLWrha2yFxVv99TK+GEPNg2CiGPe5FkfF3joMaBN31/jEUWSISXw0A830jetFSdX 5y1cBfsg/p7ga4b/VA9wo6xMwK/QJ1GbuzW7++NTggzAJhEcM0MGzZ5OT6JBIA63R8yJ In7pmNCAsPp3GWdjuDG9v3jNG3KPxQzIHPSoAPBrrm8dBIo7hAeWIlfbZSDoO9prZOHB vblCGAcDH1xAGVVpphHG4aJmvQZ1uk9n3JVHA8T0rTaFaFwtPZa0m/uXLxSSg4qTVLK1 N7SQ== X-Gm-Message-State: AOAM532M4aGk4a78mmHtwnMDKzg9JQKYAIfFNdfoH2fGDI9EsX+lSnR+ 9rUcUeIFjOcum39aXhWcN/8EPw== X-Google-Smtp-Source: ABdhPJy5RqaCyL0gNsOVmEnN6ysZO34tX6YW8ngPFDN0VQPcc0eO6AiysiwMy/x/2DZY45Q/e5yk+g== X-Received: by 2002:a17:90b:3750:: with SMTP id ne16mr2576285pjb.6.1597417208077; Fri, 14 Aug 2020 08:00:08 -0700 (PDT) Received: from hermes.lan (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id y10sm9623567pff.177.2020.08.14.08.00.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2020 08:00:07 -0700 (PDT) Date: Fri, 14 Aug 2020 08:00:04 -0700 From: Stephen Hemminger To: Chenbo Xia Cc: dev@dpdk.org, thomas@monjalon.net, xuan.ding@intel.com, xiuchun.lu@intel.com, cunming.liang@intel.com, changpeng.liu@intel.com, zhihong.wang@intel.com Message-ID: <20200814080004.67b7a0b5@hermes.lan> In-Reply-To: <20200814191606.26312-1-chenbo.xia@intel.com> References: <20200814191606.26312-1-chenbo.xia@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC v1 0/2] Add device emulation support in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Fri, 14 Aug 2020 19:16:04 +0000 Chenbo Xia wrote: > This series enables DPDK to be an alternative I/O device emulation librar= y of > building virtualized devices in separate processes outside QEMU. It intro= duces > a new library (librte_vfio_user), a new device class (emudev) and one pil= ot > device provider (avf_emudev) with its backend of Ethdev PMD (avfbe_ethdev= ). >=20 > *librte_vfio_user* is a server implementation of VFIO-over-socket[1] (also > known as vfio-user) which is a protocol that allows a device to be virtua= lized > in a separate process outside of QEMU.=20 >=20 > *emudev* is a device type for emulated devices. It is up to device provid= er to > choose the transport. In avf_emudev case, it uses vfio-user as transport > communicate with its client (e.g., QEMU). >=20 > *avf_emudev* is the emudev provider of AVF which is a device specificatio= n for > Intel Virtual Function cross generation. It=E2=80=99s implemented by an A= VF emudev > driver which offers a few APIs for avfbe_ethdev or app logic to operate.= =20 >=20 > *avfbe_ethdev* is a normal ethdev PMD to supply the basic I/O as backend = data > path of avf_emudev. One simple usage of avfbe_ethdev could be a para-virt= ualized > backend connected with network application logic. >=20 > Background & Motivation > ----------------------- > In order to reduce the attack surface, QEMU community is disaggregating Q= EMU by > removing part of device emulation from it. The disaggregated/multi-proces= s QEMU > is using VFIO-over-socket/vfio-user as the main transport mechanism to di= saggregate > I/O services from QEMU[2]. Vfio-user essentially implements the VFIO devi= ce model > presented to the user process by a set of messages over a unix-domain soc= ket. The > main difference between application using vfio-user and application using= vfio > kernel module is that device manipulation is based on socket messages for= vfio-user > but system calls for vfio kernel module. The vfio-user devices consist of= a generic > VFIO device type, living in QEMU, which is called the client[3], and the = core device > implementation (emulated device), living outside of QEMU, which is called= the server.=20 >=20 > With the introduction and support of vfio-user in QEMU, QEMU is explicitl= y adding > support for external emulated device and data path. We are trying to leve= rage that > and introducing vfio-user support in DPDK. By doing so, DPDK is enabled t= o be an > alternative I/O device emulation library of building virtualized devices = along with > high-performance data path in separate processes outside QEMU. It will be= easy for > hardware vendors to provide virtualized solutions of their hardware devic= es by > implementing emulated device in DPDK. >=20 > Except for vfio-user introduced in DPDK, this series also introduces the = first > emulated device implementation. That is emulated AVF device (avf_emudev) = implemented > by AVF emulation driver (avf_emudev driver). Emulated AVF device demos ho= w emulated > device could be implemented in DPDK. SPDK is also investigating to implem= ent use case > for NVMe.=20 >=20 > Design overview > --------------- >=20 > +----------------------------------------------------= --+ > | +---------------+ +---------------+ = | > | | avf_emudev | | avfbe_ethdev | = | > | | driver | | driver | = | > | +---------------+ +---------------+ = | > | | | = | > | ------------------------------------------- VDEV BU= S | > | | | = | > | +---------------+ +--------------+ = | > +--------------+ | | vdev: | | vdev: | = | > | +----------+ | | | /path/to/vfio | | avf_emudev_# | = | > | | Generic | | | +---------------+ +--------------+ = | > | | vfio-dev | | | | = | > | +----------+ | | | = | > | +----------+ | | +----------+ = | > | | vfio-user| | | | vfio-user| = | > | | client | |<---|----->| server | = | > | +----------+ | | +----------+ = | > | QEMU | | DPDK = | > +--------------+ +----------------------------------------------------= --+ >=20 > - vfio-user. Vfio-user in DPDK is referred to the vfio-user protocol impl= ementation > playing server role. It provides transport between emulated device and ge= neric VFIO > device in QEMU. Emulated device in DPDK and generic VFIO device in QEMU a= re working > together to present VFIO device model to VM. This series introduces vfio-= user > implementation as a library called librte_vfio_user which is under lib/li= brte_vfio_user. >=20 > - vdev:/path/to/vfio. It defines the emudev device and binds to vdev bus = driver. The > emudev device is defined by DPDK applications through command line as '--= vdev=3Demu_iavf, > path=3D/path/to/socket, id=3D#' in avf_emudev case. Parameters in command= line include device > name (emu_iavf) which is used to identify corresponding driver (in this c= ase, avf_emudev > driver which implements emudev device of AVF), path=3D/path/to/socket whi= ch is used to open > the transport interface to vfio-user client in QEMU, and id which is the = index of emudev > device. >=20 > - avf_emudev driver. It implements emulated AVF device which is the emude= v provider of > AVF. The avf_emudev_driver offers a few APIs implementation exposed by em= udev device APIs > for avfbe_ethdev_pmd or application logic to operate. These APIs are desc= ribed in > lib/librte_emudev/rte_emudev.h. >=20 > - vdev: avf_emudev_#. The vdev device is defined by DPDK application thro= ugh command line > as '--vdev=3Dnet_avfbe,id=3D#,avf_emu_id=3D#'.It is associated with emude= v provider of AVF by > 'avf_emu_id=3D#'. > =20 > - avfbe_ethdev driver. It is a normal ethdev PMD to supply the basic I/O = as backend data > path of avf_emudev. >=20 > Why not rawdev for emulated device > ---------------------------------- > Instead of introducing new class emudev, emulated device could be present= ed as rawdev. > However, existing rawdev APIs cannot meet the requirements of emulated de= vice. There are > three API categories for emudev. They are emudev device lifecycle managem= ent, backend > facing APIs, and emudev device provider facing APIs respectively. Existin= g rawdev APIs > could only cover lifecycle management APIs and some of backend facing API= s. Other APIs, > even if added to rawdev API are not required by other rawdev applications. > =20 > References > ---------- > [1]: https://patchew.org/QEMU/1594913503-52271-1-git-send-email-thanos.ma= katos@nutanix.com/ > [2]: https://wiki.qemu.org/Features/MultiProcessQEMU > [3]: https://github.com/elmarco/qemu/blob/wip/vfio-user/hw/vfio/libvfio-u= ser.c >=20 > Chenbo Xia (2): > vfio_user: Add library for vfio over socket > emudev: Add library for emulated device >=20 > lib/librte_emudev/rte_emudev.h | 315 +++++++++++++++++++++++++ > lib/librte_vfio_user/rte_vfio_user.h | 335 +++++++++++++++++++++++++++ > 2 files changed, 650 insertions(+) > create mode 100644 lib/librte_emudev/rte_emudev.h > create mode 100644 lib/librte_vfio_user/rte_vfio_user.h >=20 This looks good, but it would be good to have an example or way to integrate it into a test framework. One of the agree upon principles by the tech boa= rd is "all new features should have test cases". There have been a lot of exce= ptions though.