From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3E3BDA0350 for ; Wed, 1 Jul 2020 21:29:24 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id ABAA51BF76; Wed, 1 Jul 2020 21:29:23 +0200 (CEST) Received: from mail-vs1-f67.google.com (mail-vs1-f67.google.com [209.85.217.67]) by dpdk.org (Postfix) with ESMTP id F298F1C07E for ; Wed, 1 Jul 2020 21:29:22 +0200 (CEST) Received: by mail-vs1-f67.google.com with SMTP id v1so13930501vsb.10 for ; Wed, 01 Jul 2020 12:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infinite-io.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=BzF41X7xO4j3LV+2rRS/rIaLeWfGd/n4T+keUdr7epI=; b=Vtny9npEP2YJ6AZpMZ6bmd+dFP84k6F3a6IBLLtJZPO+U46LPMydDaV7EMSS7p0D82 EL68bsBvNeVsfmHBV+fN/4aaI8zlqT1UYnk98fdfA2WhG4qIU9rLVLYKvPVdKROSNIqD nDPlNo3LF52ph6Q6db0a3h+J7oXqYb3BCiTbBcXNbhiJy5Z5Zfs/ctb/Gh2AB87Cywdm aH6nZBEJzIJFe0MfMbcEPqg1ajIjvdlZroOBI9LJiKSC/SqTdbJQqb5O6ZsjHODrimNa 8m/7WEZWGIHnmMGV1XvXQoz1JShv/sKTEa8oHIPHhQ56VgcFv297MA6JojUj/fU8w+yJ 1gqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=BzF41X7xO4j3LV+2rRS/rIaLeWfGd/n4T+keUdr7epI=; b=iPB6X+TYNExylZE6h94D7uC5Xc8+Q8wZeDIbNFfyfRPyH+2yBEJ7xpUEs6zPgoTPnv UALQelBFOSVKr61SU8m8U+00L1YZ8hM8UUYOuCMoUhFQa3gIAsdTf/AgXcMigAqJLj/+ goBrtlS3gb6+n/OZHBWK2vd7vEShwKYYQXUQD+EkrMTIydA9/qIui0r5aajqqOSEMBbF gf/acHkaeUPFEYTobgdyb/OejwzV/xmo0MS1oway898QKs/gdzropaI0Wxji1kkWFcs0 72xpf5JD3Rv6bQI63eBr7cYLwSkCFDqx/ULOIasb5+JalyWk0hqVBm6A5a+wnL40o1v6 +8JA== X-Gm-Message-State: AOAM530sNNsQi2H2TVrtfJwrVOqZoEV9rmdA/jjUKmiKZaP/9mTDtGLp FC/blD1sLWAjmpUwYFuX9GG21FLmJ2n3Nz2jJkRyIdcA X-Google-Smtp-Source: ABdhPJz+LfrKpiPl/zWrBVzauzeRQXdzmzh2dk94SKcJfN8XZ1K+/MZkIeJ705LqHdJNG4SWdxwyQS8P6PRwQM9Lu48= X-Received: by 2002:a05:6102:3188:: with SMTP id c8mr5599677vsh.61.1593631762047; Wed, 01 Jul 2020 12:29:22 -0700 (PDT) MIME-Version: 1.0 From: Matt Laswell Date: Wed, 1 Jul 2020 14:29:11 -0500 Message-ID: To: users Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] How to wring performance out of virtio? X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hey everybody, My application relies on some control plane apps that need to talk to the network through ports that are controlled by DPDK. Historically, we've used KNI for that, which lets the control plane applications use the normal Linux network stack and interact with it with common tools. Recently, I've been looking into leaving KNI and moving to virtio for the same purpose. However, I find that I can't move traffic through it as quickly as I could with KNI. Specifically, the relevant portions of my dataplane application move data around roughly like this: NIC<->PMD<->packet ingestor (recognizes control plane traffic) <-> KNI/virtio thread <-> KNI/virtio What I find is that if my control plane application is mostly sending traffic out to the network, I can achieve roughly the same bandwidth through virtio that I did through KNI. If I'm primarily reading from the network, however, I only get about half of the bandwidth I used to get from KNI. Also, I see congestion when enqueueing for virtio. The congestion isn't alleviated by increasing queue sizes, so I presume I'm not dealing with occasional bursts, but general slowness. Obviously, optimization is a complex subject, and I'm only describing a small corner of a more complex codebase. But, for those of you who have more experience with virtio under DPDK than I do, if you were presented with a problem like this, what is the handful of questions you ask, or big wins you'd seek out first? A few details about my application: - Physical hardware, with isolated CPU cores for the dataplane, not a VM - We're using the virtio library that comes with Ubuntu, not a custom/DPDK one - Checksum generation and validation are enabled at the NIC, but TSO and LRO are not, as they don't play nicely with the rest of our application Thanks in advance, - Matt