From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qg0-f48.google.com (mail-qg0-f48.google.com [209.85.192.48]) by dpdk.org (Postfix) with ESMTP id D93152B96 for ; Thu, 21 Apr 2016 16:04:23 +0200 (CEST) Received: by mail-qg0-f48.google.com with SMTP id f74so41554862qge.2 for ; Thu, 21 Apr 2016 07:04:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BrGIkMfn+L07ZyfXD0YsPX1CxOo6EPjCYOUHg74HNu8=; b=RsUQr4i3pkw3kbY16Atra6x6DDwGrWNLW0CpKSu6uTENgk/Gg96jxqYSUBKeYXkio/ BlX4ADnstxGoQyJ2U5xbvH0WVukZ3ABPoZ3rWcDpm1s7fRkxgHDrBasLU+6pM3ruGV43 CVw2Hljo6lCqTsNhwDx2Hz0ySJhuyfApwOUZuhvMMW1hBT5wVqoyC8LXV6TxoLBWiB1p rxYpVl5KsTDJI+ovcXldaZ2N9bm2igsUbOBBGbrc3gxB5it5urxyHeQHf4v6HbAaJTyd QnHhGb81XoVGnYBTcgIEWauD8I9CYhzUdrSgFfhroTGzVWJgyNlIhZSvKeGuiwHZcQVN aH+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BrGIkMfn+L07ZyfXD0YsPX1CxOo6EPjCYOUHg74HNu8=; b=I/w6/JxqoP4moLCwjGouZNh7SbWi+eNK+UQJJI2NnS3hqimjSAV+yJKsAvm3NRKpoP Tf2/4EONhQRNChupaaWu7Rit0xwkC3bdMvQAxvdPpzTNjGqcR9Gdo1zgkvlJVefg2BQh DLSYQbU5dTGKFb+dEh+5AR0tU9yFU9fxwgiG2955XkWqLbt+nMDNxawmfVsB8XW9mrMp AQD05PHXcXmc69ghXp+HXZwwtyLqLC6NDNE8hLOb9i0BhP2vUtrn8cnCMQFv2C0vuxLP viZETTHX0TLpD0R0DuONwOxlOv794uVSJRkCWMMrlLvLRFHuSn3nPq/HxZqlKt5JNomi 1HEw== X-Gm-Message-State: AOPr4FU0gGndhuvWafdQeJZZ2my0jYZnfVNuYomkRWDFvyZtHX8k6K5K9zuzImuoH815MWEcnDriPH6Sltg/K13b X-Received: by 10.140.156.138 with SMTP id c132mr18075291qhc.96.1461247463289; Thu, 21 Apr 2016 07:04:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.212.91 with HTTP; Thu, 21 Apr 2016 07:04:03 -0700 (PDT) In-Reply-To: <5718B306.5070801@samsung.com> References: <5718B306.5070801@samsung.com> From: Christian Ehrhardt Date: Thu, 21 Apr 2016 16:04:03 +0200 Message-ID: To: Ilya Maximets Cc: dev , Daniele Di Proietto , "dev@openvswitch.org" , "yuanhan.liu@linux.intel.com" , Dyasly Sergey Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Memory leak when adding/removing vhost_user ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Apr 2016 14:04:24 -0000 Thanks Ilya, yeah we usually wait for the point releases as they undergo some extra testing and verification. .1 shouldn't be too much into the future I guess. Thanks a lot for identifying. That said, I'd still go on with Yuanhan to finalize the dpdk side leak fix we identified, so we eventually get it committed. So Yuanhan, what do you think of my last revised version of your patch for upstream DPDK (there with the vhost_destroy_device then)? I mean it is essentially your patch plus a bit of polishing, not mine so I don't feel entitled to submit it as mine :-) Kind Regards, Christian Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd On Thu, Apr 21, 2016 at 1:01 PM, Ilya Maximets wrote: > Hi, Christian. > You're, likely, using tar archive with openvswitch from openvswitch.org. > It doesn't contain many bug fixes from git/branch-2.5 unfortunately. > > The problem that you are facing has been solved in branch-2.5 by > > commit d9df7b9206831631ddbd90f9cbeef1b4fc5a8e89 > Author: Ilya Maximets > Date: Thu Mar 3 11:30:06 2016 +0300 > > netdev-dpdk: Fix memory leak in netdev_dpdk_vhost_destruct(). > > Fixes: 4573fbd38fa1 ("netdev-dpdk: Add vhost-user multiqueue support") > Signed-off-by: Ilya Maximets > Acked-by: Flavio Leitner > Acked-by: Daniele Di Proietto > > Best regards, Ilya Maximets. > > > I assume there is a leak somewhere on adding/removing vhost_user ports. > > Although it could also be "only" a fragmentation issue. > > > > Reproduction is easy: > > I set up a pair of nicely working OVS-DPDK connected KVM Guests. > > Then in a loop I > > - add up to more 512 ports > > - test connectivity between the two guests > > - remove up to 512 ports > > > > Depending on memory and the amount of multiqueue/rxq I use it seems to > > slightly change when exactly it breaks. But for my default setup of 4 > > queues and 5G Hugepages initialized by DPDK it always breaks at the sixth > > iteration. > > Here a link to the stack trace indicating a memory shortage (TBC): > > https://launchpadlibrarian.net/253916410/apport-retrace.log > > > > Known Todos: > > - I want to track it down more, and will try to come up with a non > > openvswitch based looping testcase that might show it as well to simplify > > debugging. > > - in use were Openvswitch-dpdk 2.5 and DPDK 2.2; Retest with DPDK 16.04 > and > > Openvswitch master is planned. > > > > I will go on debugging this and let you know, but I wanted to give a > heads > > up to everyone. > > In case this is a known issue for some of you please let me know. > > > > Kind Regards, > > Christian Ehrhardt > > Software Engineer, Ubuntu Server > > Canonical Ltd > > > > P.S. I think it is a dpdk issue, but adding Daniele on CC to represent > > ovs-dpdk as well. >