From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f177.google.com (mail-qk0-f177.google.com [209.85.220.177]) by dpdk.org (Postfix) with ESMTP id 7AFF52C13 for ; Thu, 21 Apr 2016 11:07:36 +0200 (CEST) Received: by mail-qk0-f177.google.com with SMTP id x7so23197386qkd.3 for ; Thu, 21 Apr 2016 02:07:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=/AGmaknpNzSmQN4RwgthRJYrnaYllBP0Gez4XD26wkA=; b=eg0HBBW2irV7qxNYX2C66lsjabc2tIEhO4GznR1KvTnAQWgaemeS/YZV6aLm370/ms zsf/XRIKlf3Mt7pzBHW37aWc/FQ7gK2dd+WwWXGiNyrPB/INMof4Fn3K6y1zCjNaJiao EmqgGmwf0VYG/PSOQoNiOKpC4aog3RSZmpi7NrVVWW1Pe89oP+oWGN1i4MEpxQbV4wEo TgaXL3YU+8d3G4RRreXtezP44ouH8xTpiAWkeKozz7r5dQdLJqu1m+/yDQdfe+jJsAmi uQJpvcJTWTV4GW0VMd9kkKmd53yNNex4wtbLa31sZi/Gdj3J8zVuqxSOkcxSoBOQ4cWh M9JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=/AGmaknpNzSmQN4RwgthRJYrnaYllBP0Gez4XD26wkA=; b=EZ08Q/4lGJxQIB5L2Sttfoy5a919aRlL1FHOLzREI7Cmc3aX/V3+4xX/H3Gl33y/qD S/PrqY1GG/Q7Jjm1TCvTC+Sda3wwHbK3gQycpNiWGeLVs/RYlylB8XG+aSsFFKFCBczS +Ym1xJDc9aoPsrMDuUzt7L/GPiXLl2GO24gC0Yprs5sejqQDGo8Vbt4OkoSy/q7n2Odx gmmP05BuEnNAwMNH38IlqtLhoMI/WJaAQSJGF0WtA2BN41iuwIK673qkTGJjOq2P9NEA S4MNO4MWy+GEIFDKA36LOu7U7zk7YYKJ1LWmdaOm9Mw+kbMP7STFDYwHuDFzWRML3iXg RbBA== X-Gm-Message-State: AOPr4FXDCv0pd04hr5NR+tvAbfYmHgU53A5HyoBZTAp8s17o/voI2RKzE2jaNhofXoRPW1zbmiQGEvdkljG5KOHl X-Received: by 10.55.78.70 with SMTP id c67mr18142077qkb.37.1461229655875; Thu, 21 Apr 2016 02:07:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.212.91 with HTTP; Thu, 21 Apr 2016 02:07:16 -0700 (PDT) In-Reply-To: <20160421055458.GD5872@yliu-dev.sh.intel.com> References: <20160418174650.GD2576@yliu-dev.sh.intel.com> <20160418181422.GE2576@yliu-dev.sh.intel.com> <20160420050445.GB5872@yliu-dev.sh.intel.com> <20160421055458.GD5872@yliu-dev.sh.intel.com> From: Christian Ehrhardt Date: Thu, 21 Apr 2016 11:07:16 +0200 Message-ID: To: Yuanhan Liu Cc: dev , Daniele Di Proietto Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Memory leak when adding/removing vhost_user ports X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Apr 2016 09:07:36 -0000 Hi, I can follow your argument that - and agree that in this case the leak can't be solved your patch. Still I found it useful to revise it along our discussion as eventually it will still be a good patch to have. I followed your suggestion and found: - rte_vhost_driver_register callocs vserver (implies fh=0) - later when on init when getting the callback to vserver_new_vq_conn it would get set by ops->new_device(vdev_ctx); - but as you pointed out that could be fh = 0 for the first - so I initialized vserver->fh with -1 in rte_vhost_driver_register - that won't ever be a real fh. - later on get_config_ll_entry won't find a device with that then on the call by destroy_device. - so the revised patch currently in use (still for DPDK 2.2) can be found here http://paste.ubuntu.com/15961394/ Also as you requested I tried with no guest attached at all - that way I can still reproduce it. Here is a new stacktrace, but to me it looks the same http://paste.ubuntu.com/15961185/ Also as you asked before a log of the vswitch, but it is 895MB since a lot of messages repeat on port add/remove. Even compressed still 27MB - I need to do something about verbosity there. Also the system journal of the same time. Therefore I only added links to bz2 files. The crash is at "2016-04-21T07:54:47.782Z" in the logs. => http://people.canonical.com/~paelzer/ovs-dpdk-vhost-add-remove-leak/mem-leak-addremove.journal.bz2 => http://people.canonical.com/~paelzer/ovs-dpdk-vhost-add-remove-leak/ovs-vswitchd.log.bz2 Kind Regards, Christian Ehrhardt Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd On Thu, Apr 21, 2016 at 7:54 AM, Yuanhan Liu wrote: > On Wed, Apr 20, 2016 at 08:18:49AM +0200, Christian Ehrhardt wrote: > > On Wed, Apr 20, 2016 at 7:04 AM, Yuanhan Liu < > yuanhan.liu@linux.intel.com> > > wrote: > > > > On Tue, Apr 19, 2016 at 06:33:50PM +0200, Christian Ehrhardt wrote: > > > > [...] > > > > > With that applied one (and only one) of my two guests looses > connectivity > > after > > > removing the ports the first time. > > > > Yeah, that's should be because I invoked the "->destroy_device()" > > callback. > > > > > > Shouldn't that not only destroy the particular vhost_user device I > remove? > > I assume the "not" is typed wrong here, then yes. Well, it turned > out that I accidentally destroyed the first guest (with id 0) with > following code: > > ctx.fh = g_vhost_server.server[i]->fh; > vhost_destroy_device(ctx); > > server[i]->fh is initialized with 0 when no connection is established > (check below for more info), and the first id is started with 0. Anyway, > this could be fixed easily. > > > See below for some better details on the test to clarify that. > > > > > > BTW, I'm curious how do you do the test? I saw you added 256 ports, > but > > with 2 guests only? So, 254 of them are idle, just for testing the > > memory leak bug? > > > > > > Maybe I should describe it better: > > 1. Spawn some vhost-user ports (40 in my case) > > 2. Spawn a pair of guests that connect via four of those ports per guest > > 3. Guests only intialize one of that vhost_user based NICs > > 4. check connectivity between guests via the vhost_user based connection > > (working at this stage) > > LOOP 5-7: > > 5. add ports 41-512 > > 6. remove ports 41-512 > > 7. check connectivity between guests via the vhost_user based > connection > > Yes, it's much clearer now. Thanks. > > I then don't see it's a leak from DPDK vhost-user, at least not the leak > on "struct virtio_net" I have mentioned before. "struct virito_net" will > not even be allocated for those ports never used (ports 41-512 in your > case), > as it will be allocated only when there is a connection established, aka, > a guest is connected. > > BTW, will you be able to reproduce it without any connections? Say, all > 512 ports are added, and then deleted. > > Thanks. > > --yliu > > > > > So the vhost_user ports the guests are using are never deleted. > > Only some extra (not even used) ports are added&removed in the loop to > search > > for potential leaks over a longer lifetime of an openvswitch-dpdk based > > solution. > > >