From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68D21A0350 for ; Thu, 25 Jun 2020 20:03:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id CEBCFE07; Thu, 25 Jun 2020 20:03:46 +0200 (CEST) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by dpdk.org (Postfix) with ESMTP id 8080DCF3 for ; Thu, 25 Jun 2020 20:03:45 +0200 (CEST) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 05PI20fv030459 for ; Thu, 25 Jun 2020 14:03:44 -0400 Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 31ux00huc0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 25 Jun 2020 14:03:44 -0400 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 05PI3iMU036875 for ; Thu, 25 Jun 2020 14:03:44 -0400 Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 31ux00hubp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 25 Jun 2020 14:03:44 -0400 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 05PHxg62015143; Thu, 25 Jun 2020 18:03:43 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma04dal.us.ibm.com with ESMTP id 31uurq9fxg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 25 Jun 2020 18:03:43 +0000 Received: from b03ledav006.gho.boulder.ibm.com (b03ledav006.gho.boulder.ibm.com [9.17.130.237]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 05PI3g4j50659674 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Jun 2020 18:03:42 GMT Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3EB7AC6057; Thu, 25 Jun 2020 18:03:42 +0000 (GMT) Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 04EC8C6055; Thu, 25 Jun 2020 18:03:41 +0000 (GMT) Received: from Davids-MBP.randomparity.org (unknown [9.211.152.140]) by b03ledav006.gho.boulder.ibm.com (Postfix) with ESMTP; Thu, 25 Jun 2020 18:03:41 +0000 (GMT) To: Vipul Ujawane , users@dpdk.org References: From: David Christensen Message-ID: Date: Thu, 25 Jun 2020 11:03:41 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.687 definitions=2020-06-25_12:2020-06-25, 2020-06-25 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 suspectscore=0 clxscore=1011 bulkscore=0 cotscore=-2147483648 mlxlogscore=999 phishscore=0 mlxscore=0 lowpriorityscore=0 impostorscore=0 spamscore=0 priorityscore=1501 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2004280000 definitions=main-2006250109 Subject: Re: [dpdk-users] Poor performance when using OVS with DPDK X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 6/24/20 4:03 AM, Vipul Ujawane wrote: > Dear all, > > I am observing a very low performance when running OVS-DPDK when compared > to OVS running with the Kernel Datapath. > I have OvS version 2.13.90 compiled from source with the latest stable DPDK > v19.11.3 on a stable Debian system running kernel 4.19.0-9-amd64 (real > version:4.19.118). > > I have tried to use the latest released OvS as well (2.12) with the same > LTS DPDK. As a last resort, I have tried an older kernel, whether it has > any problem (4.19.0-8-amd64 (real version:4.19.98)). > > I have not been able to troubleshoot the problem, and kindly request your > help regarding the same. > > HW configuration > ================ > We have to two totally identical servers (Debian stable, Intel(R) Xeon(R) > Gold 6230 CPU, 96G Mem), each runs KVM virtual machine. On the hypervisor > layer, we have OvS for traffic routing. The servers are connected directly > via a Mellanox ConnectX-5 (1x100G). > OVS Forwarding tables are configured for simple port-forwarding only to > avoid any packet processing-related issue. > > Problem > ======= > When both servers are running OVS-Kernel at the hypervisor layer and VMs > are connected to it via libvirt and virtio interfaces, the > VM->Server1->Server2->VM throughput is around 16-18Gbps. > However, when using OVS-DPDK with the same setting, the throughput drops > down to 4-6Gbps. You don't mention the traffic profile. I assume 64 byte frames but best to be explicit. > > SW/driver configurations: > ================== > DPDK > ---- > In config common_base, besides the defaults, I have enabled the following > extra drivers/features to be compiled/enabled. > CONFIG_RTE_LIBRTE_MLX5_PMD=y > CONFIG_RTE_LIBRTE_VHOST=y > CONFIG_RTE_LIBRTE_VHOST_NUMA=y > CONFIG_RTE_LIBRTE_PMD_VHOST=y > CONFIG_RTE_VIRTIO_USER=n > CONFIG_RTE_EAL_VFIO=y > > > OVS > --- > $ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.13.90 > > $sudo ovs-vsctl get Open_vSwitch . dpdk_initialized > true > > $sudo ovs-vsctl get Open_vSwitch . dpdk_version > "DPDK 19.11.3" > > OS settings > ----------- > $ lsb_release -a > No LSB modules are available. > Distributor ID: Debian > Description: Debian GNU/Linux 10 (buster) > Release: 10 > Codename: buster > > > $ cat /proc/cmdline > BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable > ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt > quiet Why don't you reserve any CPUs for OVS/DPDK or VM usage? All published performance white papers recommend settings for CPU isolation like this Mellanox DPDK performance report: https://fast.dpdk.org/doc/perf/DPDK_19_08_Mellanox_NIC_performance_report.pdf For their test system: isolcpus=24-47 intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable nohz_full=24-47 rcu_nocbs=24-47 rcu_nocb_poll default_hugepagesz=1G hugepagesz=1G hugepages=64 audit=0 nosoftlockup Using the tuned service (CPU partitioning profile) make this process easier: https://tuned-project.org/ > > ./usertools/dpdk-devbind.py --status > Network devices using kernel driver > =================================== > 0000:b3:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens2 drv=mlx5_core > unused=igb_uio,vfio-pci > > Due to the way how Mellanox cards and their driver work, I have not bond > igb_uio to the interface, however, uio, igb_uio and vfio-pci kernel modules > are loaded. > > > Relevant part of the VM-config for Qemu/KVM > ------------------------------------------- > > 4096 > > Where did you get these CPU mapping values? x86 systems typically map even-numbered CPUs to one NUMA node and odd-numbered CPUs to a different NUMA node. You generally want to select CPUs from the same NUMA node as the mlx5 NIC you're using for DPDK. You should have at least 4 CPUs in the VM, selected according to the NUMA topology of the system. Take a look at this bash script written for Red Hat: https://github.com/ctrautma/RHEL_NIC_QUALIFICATION/blob/ansible/ansible/get_cpulist.sh It gives you a good starting reference which CPUs to select for the OVS/DPDK and VM configurations on your particular system. Also review the Ansible script pvp_ovsdpdk.yml, it provides a lot of other useful steps you might be able to apply to your Debian OS. > > > > > > > memAccess='shared'/> > > > > > mo$ > > > Is there a requirement for mergeable RX buffers? Some PMDs like mlx5 can take advantage of SSE instructions when this is disabled, yielding better performance. > >
function='0x0'$ > > I don't see hugepage usage in the libvirt XML. Something similar to: 8388608 8388608 > ----------------------------------- > OVS Start Config > ----------------------------------- > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,0" > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0xff > ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0e These two masks shouldn't overlap: https://developers.redhat.com/blog/2017/06/28/ovs-dpdk-parameters-dealing-with-multi-numa/ > ovs-vsctl add-port ovsbr dpdk0 -- set Interface dpdk0 type=dpdk > options:dpdk-devargs=0000:b3:00.0 > ovs-vsctl set interface dpdk0 options:n_rxq=2 > ovs-vsctl add-port ovsbr vhost-vm -- set Interface vhostuser > type=dpdkvhostuser > > > > ------------------------------------------------------- > $cat /proc/cmdline > BOOT_IMAGE=/vmlinuz-4.19.0-9-amd64 root=/dev/mapper/Volume0-debian--stable > ro default_hugepagesz=1G hugepagesz=1G hugepages=16 intel_iommu=on iommu=pt > quiet > > > Is there anything I should be aware of the versions and setting I am using? > Did I compile DPDK and/or OvS in a wrong way? > > Thank you for your kind help ;) >