From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8C641A0350 for ; Wed, 24 Jun 2020 13:03:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 3D01A1D8EA; Wed, 24 Jun 2020 13:03:51 +0200 (CEST) Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by dpdk.org (Postfix) with ESMTP id 4CD9F1D716 for ; Wed, 24 Jun 2020 13:03:49 +0200 (CEST) Received: by mail-wr1-f67.google.com with SMTP id b6so1810289wrs.11 for ; Wed, 24 Jun 2020 04:03:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=xL40871DwjEAdfxS6KQaoNQZIFSTRAqXgCSXSXWhvek=; b=TTjtid3Q75Fd/aREP67pTE3r9l5gQYa2KT2hEHXVO/eNe5IDYhw+42p2qnYoflLlwZ rgiBqDRZuooL104VEgOFDAZy0wtETvB00yh2aWrkC8OBMdN3XDo87pHuIgKRWJfGsJ4z y7tzx7t96kI+pvCVw/WZjWzqIngvq18k1vAHPtm0AHxtzHKbAzk2F9gdyj0wRk1hT4cv PUugQ5zTuLP/sKBbIFUVn528NhHfTUM5ADCdv3mOH4jwFypL3j/yk+gmayZAjbr3NwmM hNH5OYTDLbNXF5ziqacILPoJrppB0DNLMAhTW7NHOnzZoPczm3yiZrrddqHQeHnbMpJM AyOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xL40871DwjEAdfxS6KQaoNQZIFSTRAqXgCSXSXWhvek=; b=K+9eMB4TmOnWosa5JDROvN6T5qP+EOkD+KXpJv2JS52CesQnU+WIVUnec/FbbsljA7 xyAaU2YaxRf8zKVcPXoI61SatQjMix6UNke6P2MFtpjQTOmuX528XnZtrWNP8ukbANHf PqooVUFjBXYfuubUKWp9MNhZ5aWhhsMIPP1KUMUraoXuFFy3wGZiDIZm8oeBJ4Q9qlh1 2rbecIEU2Mg+RTrTeKyv73jbTBa1PQx8r7Ugq9hyTnK1vJMjSUJ7L6M/WhIMnRIQaiLr SlJpx9CUDJBigYIiF+dcbWi+j3rhPS8Ifkg1fBw5y1ysrnOpVwa7/4giTzc+Aj5/9OK0 fiDg== X-Gm-Message-State: AOAM5303X8SIDNk8x1q2VrWaqJcT72SlT3o+NJjHRI130pJ1EkfkCBj6 VDy3+kGEZ1y8KRFaISVgSU0L/uehwRmkCdHlpHmvTVjqkds= X-Google-Smtp-Source: ABdhPJxVPpGuIaGCC6fDB+8x6CjqQtTgzlGpbpKKdF/jDLTJVtfoBuvgr0dOOtfz/GRTzUsl8ufpnbItxdqx6txabcI= X-Received: by 2002:a5d:630d:: with SMTP id i13mr26716786wru.208.1592996628404; Wed, 24 Jun 2020 04:03:48 -0700 (PDT) MIME-Version: 1.0 From: Vipul Ujawane Date: Wed, 24 Jun 2020 19:03:09 +0800 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Poor performance when using OVS with DPDK X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Dear all, I am observing a very low performance when running OVS-DPDK when compared to OVS running with the Kernel Datapath. I have OvS version 2.13.90 compiled from source with the latest stable DPDK v19.11.3 on a stable Debian system running kernel 4.19.0-9-amd64 (real version:4.19.118). I have tried to use the latest released OvS as well (2.12) with the same LTS DPDK. As a last resort, I have tried an older kernel, whether it has any problem (4.19.0-8-amd64 (real version:4.19.98)). I have not been able to troubleshoot the problem, and kindly request your help regarding the same. HW configuration ================ We have to two totally identical servers (Debian stable, Intel(R) Xeon(R) Gold 6230 CPU, 96G Mem), each runs KVM virtual machine. On the hypervisor layer, we have OvS for traffic routing. The servers are connected directly via a Mellanox ConnectX-5 (1x100G). OVS Forwarding tables are configured for simple port-forwarding only to avoid any packet processing-related issue. Problem ======= When both servers are running OVS-Kernel at the hypervisor layer and VMs are connected to it via libvirt and virtio interfaces, the VM->Server1->Server2->VM throughput is around 16-18Gbps. However, when using OVS-DPDK with the same setting, the throughput drops down to 4-6Gbps. SW/driver configurations: ================== DPDK ---- In config common_base, besides the defaults, I have enabled the following extra drivers/features to be compiled/enabled. CONFIG_RTE_LIBRTE_MLX5_PMD=y CONFIG_RTE_LIBRTE_VHOST=y CONFIG_RTE_LIBRTE_VHOST_NUMA=y CONFIG_RTE_LIBRTE_PMD_VHOST=y CONFIG_RTE_VIRTIO_USER=n CONFIG_RTE_EAL_VFIO=y OVS --- $ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.13.90 $sudo ovs-vsctl get Open_vSwitch . dpdk_initialized true $sudo ovs-vsctl get Open_vSwitch . dpdk_version "DPDK 19.11.3" OS settings ----------- $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster $ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt quiet ./usertools/dpdk-devbind.py --status Network devices using kernel driver =================================== 0000:b3:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens2 drv=mlx5_core unused=igb_uio,vfio-pci Due to the way how Mellanox cards and their driver work, I have not bond igb_uio to the interface, however, uio, igb_uio and vfio-pci kernel modules are loaded. Relevant part of the VM-config for Qemu/KVM ------------------------------------------- 4096
----------------------------------- OVS Start Config ----------------------------------- ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,0" ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xff ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0e ovs-vsctl add-port ovsbr dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:b3:00.0 ovs-vsctl set interface dpdk0 options:n_rxq=2 ovs-vsctl add-port ovsbr vhost-vm -- set Interface vhostuser type=dpdkvhostuser ------------------------------------------------------- $cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt quiet Is there anything I should be aware of the versions and setting I am using? Did I compile DPDK and/or OvS in a wrong way? Thank you for your kind help ;) -- Vipul Ujawane