From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3BEEAA0350 for ; Tue, 30 Jun 2020 09:33:30 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B48061BF7B; Tue, 30 Jun 2020 09:32:47 +0200 (CEST) Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by dpdk.org (Postfix) with ESMTP id 68E1C1D519 for ; Wed, 24 Jun 2020 12:57:22 +0200 (CEST) Received: by mail-wm1-f65.google.com with SMTP id l2so588519wmf.0 for ; Wed, 24 Jun 2020 03:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=zrrDt2MpIRYQ4Z9AQzGAJuV9qNjdRShox6lMn2X/wqk=; b=Y4bOQ9cr0c9TUWEskxxipirdbT2UZSoTQvK1p3AisAWtqrd5hCBeEEUo77t5uwnNrO 7mv2enEyNjdiyUAXRurABqnWKBOM4AGwziVYZXnjlFVsTA5PZsXG6aWIslJJSGnedPoe hXj8eP/zSyNgOKPOuL/U2jgJ/LV8cLe9kwnU3fZ5FkmcDceBCvuEDMgCZm9GFhPbjccK aW9DEckL2BuBySmiMfwzGjsVuTdXeNFa6L+Xj7fVUjKJcsvwxYYRc/EvRrUP5Q7g1e6C gA+041AHTkDRv46lAcEfkQvZCjvihxHKOWE2vyZc122SQ3BmAWzQlEWfQS2nQA+/q3ze KT8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=zrrDt2MpIRYQ4Z9AQzGAJuV9qNjdRShox6lMn2X/wqk=; b=l3Bp1An1s2T1TXlQ9KsnshlqxcpOgJ21adijMsDwuIbkr78dSuKMLQw9jGYSWvANsS 9FuXzTjxGTa4hMZy9TJD/NWI0BRdqBvbTCsxo0sq/M30cig5dd06pbt1bSOFTlZwCO0F lBmUKNHtzg5wtTsP9K1XtOhL+zNVhiisbOvkBv4rzqzNX+JbmWZUZKq6/bshGsyopF9n 5avSpUKDZvZGDey1H0YGLz0kGFpg8RKHkurDM/97pU/i29pHtQ7UaBhJtXotk2TzlH0D FNODNm2OtMQsoofcQpBNUuZw+qQQYO3ObKzk/Nn1DRKiAlptYhVSlsWjBAnmh1wO0nIT 5q1Q== X-Gm-Message-State: AOAM533xrfn5pwEUqRUmH8k1jJDy8J3hKSVixKWssBQTJIVswH4fAF2c qLrwWLokGcvdKaokqOZsB+FlP82bHUlNoEqxrOVma5WxCpQ= X-Google-Smtp-Source: ABdhPJyd5eh4SO1dlZt3T5cdg3ca60EnvUqqkisNkCG39rBC5VqlDNhd/wfLBbnRNVOd44nkONklx+kFxrRmSnTDyRg= X-Received: by 2002:a05:600c:2f17:: with SMTP id r23mr27958950wmn.167.1592996241974; Wed, 24 Jun 2020 03:57:21 -0700 (PDT) MIME-Version: 1.0 From: Vipul Ujawane Date: Wed, 24 Jun 2020 18:56:43 +0800 Message-ID: To: ovs-discuss@openvswitch.org X-Mailman-Approved-At: Tue, 30 Jun 2020 09:32:46 +0200 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Poor performance when using OVS with DPDK X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Dear all, I am observing a very low performance when running OVS-DPDK when compared to OVS running with the Kernel Datapath. I have OvS version 2.13.90 compiled from source with the latest stable DPDK v19.11.3 on a stable Debian system running kernel 4.19.0-9-amd64 (real version:4.19.118). I have tried to use the latest released OvS as well (2.12) with the same LTS DPDK. As a last resort, I have tried an older kernel, whether it has any problem (4.19.0-8-amd64 (real version:4.19.98)). I have not been able to troubleshoot the problem, and kindly request your help regarding the same. HW configuration ================ We have to two totally identical servers (Debian stable, Intel(R) Xeon(R) Gold 6230 CPU, 96G Mem), each runs KVM virtual machine. On the hypervisor layer, we have OvS for traffic routing. The servers are connected directly via a Mellanox ConnectX-5 (1x100G). OVS Forwarding tables are configured for simple port-forwarding only to avoid any packet processing-related issue. Problem ======= When both servers are running OVS-Kernel at the hypervisor layer and VMs are connected to it via libvirt and virtio interfaces, the VM->Server1->Server2->VM throughput is around 16-18Gbps. However, when using OVS-DPDK with the same setting, the throughput drops down to 4-6Gbps. SW/driver configurations: ================== DPDK ---- In config common_base, besides the defaults, I have enabled the following extra drivers/features to be compiled/enabled. CONFIG_RTE_LIBRTE_MLX5_PMD=y CONFIG_RTE_LIBRTE_VHOST=y CONFIG_RTE_LIBRTE_VHOST_NUMA=y CONFIG_RTE_LIBRTE_PMD_VHOST=y CONFIG_RTE_VIRTIO_USER=n CONFIG_RTE_EAL_VFIO=y OVS --- $ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.13.90 $sudo ovs-vsctl get Open_vSwitch . dpdk_initialized true $sudo ovs-vsctl get Open_vSwitch . dpdk_version "DPDK 19.11.3" OS settings ----------- $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster $ cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt quiet ./usertools/dpdk-devbind.py --status Network devices using kernel driver =================================== 0000:b3:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens2 drv=mlx5_core unused=igb_uio,vfio-pci Due to the way how Mellanox cards and their driver work, I have not bond igb_uio to the interface, however, uio, igb_uio and vfio-pci kernel modules are loaded. Relevant part of the VM-config for Qemu/KVM ------------------------------------------- 4096
----------------------------------- OVS Start Config ----------------------------------- ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,0" ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xff ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0e ovs-vsctl add-port ovsbr dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:b3:00.0 ovs-vsctl set interface dpdk0 options:n_rxq=2 ovs-vsctl add-port ovsbr vhost-vm -- set Interface vhostuser type=dpdkvhostuser ------------------------------------------------------- $cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt quiet Is there anything I should be aware of the versions and setting I am using? Did I compile DPDK and/or OvS in a wrong way? Thank you for your kind help ;) -- Vipul Ujawane