From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by dpdk.org (Postfix) with ESMTP id F063B106A for ; Tue, 17 Jan 2017 08:09:49 +0100 (CET) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP; 16 Jan 2017 23:09:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,243,1477983600"; d="scan'208";a="1113750078" Received: from dpdk06.sh.intel.com ([10.239.129.195]) by fmsmga002.fm.intel.com with ESMTP; 16 Jan 2017 23:09:47 -0800 From: Jianfeng Tan To: dev@dpdk.org Cc: yuanhan.liu@linux.intel.com, stephen@networkplumber.org, lei.a.yao@intel.com, Jianfeng Tan Date: Tue, 17 Jan 2017 07:10:20 +0000 Message-Id: <1484637030-106261-1-git-send-email-jianfeng.tan@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1480810702-114815-1-git-send-email-jianfeng.tan@intel.com> References: <1480810702-114815-1-git-send-email-jianfeng.tan@intel.com> Subject: [dpdk-dev] [PATCH v4 00/10] rxq interrupt mode for virtio PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jan 2017 07:09:50 -0000 v4: - Update documents: * doc/guides/nics/features/virtio.ini * doc/guides/nics/features/virtio_vec.ini * doc/guides/nics/virtio.rst - Remove unneeded parenthesis in return statement. - Change c++ comment style. v3: - Use hw->max_queue_pairs instead of dev->data->nb_rx_queues to allocate intr_vec array. - Fix v2 not working on legacy virtio devices by moving msix enabling before queue/irq binding. - Reword cover letter to give an overview of this series. - Remove wrapper to call vtpci->set_config_irq and vtpci->set_queue_irq. - Rebase on the new code, and fix a bug after changes by the commit bb30369dc10("eal: allow passing const interrupt handle"). Basically, it changes the way to get max interrupts. And we need to re-register callback to update intr_handle->max_intr. - In l3fwd-power ptype fix, use rte_eth_dev_get_supported_ptypes() to query if PMD provides needed ptypes. v2: - Add PCI queue/irq config ops. - Move rxq interrupt settings before sending DRIVER OK. Historically, virtio PMD can only be binded to igb_uio or uio_pci_generic, and not for vfio-pci (iommu group cannot be created as vIOMMU is not enabled in QEMU yet). Besides, quote from http://dpdk.org/doc/guides-16.11/rel_notes/release_2_1.html: "Per queue RX interrupt events are only allowed in VFIO which supports multiple MSI-X vectors." Linux starts to support VFIO NO-IOMMU mode since 4.8.0. It cannot put devices into groups for separation as normal VFIO does. So it does not require QEMU to support vIOMMU. But it does inherit other benefits from VFIO framework, like better interrupts supports (than UIO). It gives a good chance to enable rxq interrupt for virtio PMD. To implement it, a. Firstly, we need to enable msix. This should be done before DRIVER_OK setting and also before queue/irq binding in step b. b. Bind queue/irq through portio (legacy devices) or mmio (modern devices). So far, we hard-code 1:1 queue/irq mapping (each rx queue has one exclusive interrupt), like this: vec 0 -> config irq vec 1 -> rxq0 vec 2 -> rxq1 ... which means, the "vectors" option of QEMU should be configured with a value >= N+1 (N is the number of the queue pairs). c. To enable/disable interrupt notification, flags on virtqueues are used to control devices either sending interrupt or not. d. Encap above behaviors into callbacks in ether_dev_ops, like rx_queue_intr_enable/rx_queue_intr_disable/rx_descriptor_done etc. How to test: Step 1, prepare a VM image with kernel version >= 4.8.0, and make sure the kernel is compiled with CONFIG_VFIO_NOIOMMU=y. Step 2, on the host, start a testpmd with a vhost port: $ testpmd -c 0x7 -m 1024 --vdev 'eth_vhost0,iface=/tmp/sock0,queues=2' \ --no-pci -- -i --rxq=2 --txq=2 --nb-cores=2 Step 3, boot the VM: $ qemu ... -chardev socket,id=chr1,path=/tmp/sock0 \ -netdev vhost-user,id=net1,chardev=chr1,vhostforce,queues=2 \ -device virtio-net-pci,netdev=net1,mq=on,vectors=5 ... Step 4, insert kernel modules $ modprobe vfio enable_unsafe_noiommu_mode=1 $ modprobe vfio-pci Step 5, start l3fwd-power in VM: $ l3fwd-power -c 0x3 -n 4 -- -p 1 -P --config="(0,0,1),(0,1,1)" \ --no-numa --parse-ptype Step 6, send packets from testpmd on the host: $ start tx_first Then l3fwd-power outputs: L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 0 L3FWD_POWER: lcore 1 is waked up from rx interrupt on port 0 queue 1 Signed-off-by: Jianfeng Tan Tested-by: Lei Yao Jianfeng Tan (10): net/virtio: fix rewriting LSC flag net/virtio: clean up wrapper of set_config_irq net/virtio: add Rx descriptor check net/virtio: add PCI ops for queue/irq binding net/virtio: add Rx queue intr enable/disable functions net/virtio: setup rxq interrupts net/virtio: unbind intr/eventfd when stop device net/virtio: unmapping queue/irq when close device examples/l3fwd-power: add parse-ptype option examples/l3fwd-power: fix not stop and close device doc/guides/nics/features/virtio.ini | 1 + doc/guides/nics/features/virtio_vec.ini | 1 + doc/guides/nics/virtio.rst | 58 ++++++++++++ drivers/net/virtio/virtio_ethdev.c | 161 ++++++++++++++++++++++++++++++-- drivers/net/virtio/virtio_ethdev.h | 3 + drivers/net/virtio/virtio_pci.c | 31 ++++-- drivers/net/virtio/virtio_pci.h | 5 +- drivers/net/virtio/virtio_rxtx.c | 9 ++ drivers/net/virtio/virtqueue.c | 11 --- drivers/net/virtio/virtqueue.h | 16 +++- examples/l3fwd-power/main.c | 119 ++++++++++++++++++++++- 11 files changed, 384 insertions(+), 31 deletions(-) -- 2.7.4