From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-f174.google.com (mail-ob0-f174.google.com [209.85.214.174]) by dpdk.org (Postfix) with ESMTP id DA3688D96 for ; Tue, 1 Dec 2015 18:55:32 +0100 (CET) Received: by obbnk6 with SMTP id nk6so11560212obb.2 for ; Tue, 01 Dec 2015 09:55:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PbyXVrUw3az2mX223pzgwpj6eeQEE/NUWgYDgCWQfJw=; b=tE3LV0RD37qPSYZ5WM0AsanwqOTY9lehfNrf2L7r6NOkTHJG0+FuxDGHEinC+AeBKo MEg5sESMUGJlQh6FMziurUBpU5iXYQ3MKEyMue6o6hQ9Hm2ZBQ1nk91DTzLE7XUjAbOQ Xuk6Nt+J8q5KiKsfviZZuI1/3fwvMLrt/9Ea5zovNAI3BHqdIJeVQ2VCuP4bjDid8Z9C wpc18AiVsu6iloevbsQo/Ck6TrdSe/3gUK6frmDdK0jb1bNsCGeSVANx3IkeY5iX+onT /5OmATMD3aeQKS2/F4yF/ZRJavzYuw/73uFG4ypjJhRlLLpA/l6NrKOSgaTSMXlunSbO 9WZg== MIME-Version: 1.0 X-Received: by 10.60.17.1 with SMTP id k1mr34082229oed.4.1448992532333; Tue, 01 Dec 2015 09:55:32 -0800 (PST) Received: by 10.76.0.105 with HTTP; Tue, 1 Dec 2015 09:55:32 -0800 (PST) In-Reply-To: References: Date: Tue, 1 Dec 2015 12:55:32 -0500 Message-ID: From: Kyle Larose To: Zhihan Jiang Content-Type: text/plain; charset=UTF-8 Cc: users@dpdk.org Subject: Re: [dpdk-users] DPDK weird forwarding loss X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Dec 2015 17:55:33 -0000 Hi Jack, On Mon, Nov 30, 2015 at 8:42 PM, Zhihan Jiang wrote: > Hello, > > Other settings: intel_iommu=on iommu=pt / blacklist ixgbevf on the host > machine / pause frame off / pin all the ports to the same NUMA node & > socket / VM uses CPU on the same NUMA node & socket. > > but there is always ~0.5%- 1% packet loss > > The command line for l3fwd is: > ./build/l3fwd -c6 -n4 -w [whitelist devices] --socket-mem=1024,0 -- -p3 > --config "(0,0,1), (1,0,2)" > Try isolating the CPUs on your guest and host so that the forwarding application cannot be preempted by something else. See the 'isolcpus' kernel boot parameter. http://www.linuxtopia.org/online_books/linux_kernel/kernel_configuration/re46.html In my experience, drops that happen at low rates are caused by the polling thread not being scheduled to receive packets often enough. This can either be because it was preempted, which isolcpus will fix, or because it is sleeping. IIRC l3fwd doesn't sleep, which leaves the first case the only possibility. Hope that helps, Kyle