From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <christian.ehrhardt@canonical.com>
Received: from mail-qg0-f54.google.com (mail-qg0-f54.google.com
 [209.85.192.54]) by dpdk.org (Postfix) with ESMTP id 6FB372BFE
 for <dev@dpdk.org>; Mon, 18 Apr 2016 19:18:25 +0200 (CEST)
Received: by mail-qg0-f54.google.com with SMTP id f52so120355829qga.3
 for <dev@dpdk.org>; Mon, 18 Apr 2016 10:18:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=canonical-com.20150623.gappssmtp.com; s=20150623;
 h=mime-version:from:date:message-id:subject:to:cc;
 bh=ZhNmNhGbC0QfdjU0iw3eN+xtsDDnfI+8fDcVpaZvC/E=;
 b=Fnh7v4gTwyD/sa/R1qmihWtXEOJ+ZpTSZsaSiof1e5nUHcu4UywhF2m5ow2jnT3Lok
 JO797p0lkyyLW6CiBZzecffWW57ix4l6BwghVptJvtS/BSJUOWP1Oam5qWey9mdc+bBy
 AZM1AhpvwLMbbVcfs/0Qno9oqtlGLZ9Hghdv6Dj/ooQ19b/i7KkGqccyYUtFvEotY6bM
 va1arj3VSCPlM2+8VIeRjWScjzTExq+UfvQo1Fcrc8GUC0AunVY+xx9kcf8FMsdNQgVG
 tq/wVvDZ4qKUIiDe5ztvHu4qI3WEL4CK7SJj/g+CRYZbtqQtSsq/LdJxNPDoGjSw3Uf9
 5uVA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc;
 bh=ZhNmNhGbC0QfdjU0iw3eN+xtsDDnfI+8fDcVpaZvC/E=;
 b=UKl9zphZnVQt89hPLDSgBeLQEuoxSWDaJtmXMw31UgfGGQEl+hikhdvfrguMM8UCGr
 pjMqoVUE8JMpC6N3NpuVr0BvYjf5SsxldNxMVSKxmLaKpaxICS7WoWGNFp5y0FKhTINT
 UdQlOBkhhSBv0xmPCiS08FOaEGW3JKB/ZrHb3/VfCnQlaM0cBwNtjsdpC1fJWvLWwXGD
 R5N6bEM8T9q1rXENPnRaRcLrXXev+BOeTtvXWnfsCkuopG/A2ZlKZ7Wk/Gi5+9npagQ6
 rUYXKXo5Z6X+I3lO008BW3Nt8W2j0I4BwAorlDNviKwTas343093bzNTCUcQ5cv9x227
 Ivyw==
X-Gm-Message-State: AOPr4FVTzB8wiCilUU9baeapW8MwQf1lznjB0RsMiKLKK04atEsLYLVjz8ZCGJZjIDiQf5q6MJTEwystQwQaVe91
X-Received: by 10.55.79.17 with SMTP id d17mr45793578qkb.56.1460999904773;
 Mon, 18 Apr 2016 10:18:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.55.212.91 with HTTP; Mon, 18 Apr 2016 10:18:05 -0700 (PDT)
From: Christian Ehrhardt <christian.ehrhardt@canonical.com>
Date: Mon, 18 Apr 2016 19:18:05 +0200
Message-ID: <CAATJJ0JLd9iqNBvD=kyfpzk5DjGF_pGz+hBNeionZpuctNBymQ@mail.gmail.com>
To: dev <dev@dpdk.org>
Cc: Daniele Di Proietto <diproiettod@vmware.com>
Content-Type: text/plain; charset=UTF-8
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: [dpdk-dev] Memory leak when adding/removing vhost_user ports
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2016 17:18:25 -0000

I assume there is a leak somewhere on adding/removing vhost_user ports.
Although it could also be "only" a fragmentation issue.

Reproduction is easy:
I set up a pair of nicely working OVS-DPDK connected KVM Guests.
Then in a loop I
   - add up to more 512 ports
   - test connectivity between the two guests
   - remove up to 512 ports

Depending on memory and the amount of multiqueue/rxq I use it seems to
slightly change when exactly it breaks. But for my default setup of 4
queues and 5G Hugepages initialized by DPDK it always breaks at the sixth
iteration.
Here a link to the stack trace indicating a memory shortage (TBC):
https://launchpadlibrarian.net/253916410/apport-retrace.log

Known Todos:
- I want to track it down more, and will try to come up with a non
openvswitch based looping testcase that might show it as well to simplify
debugging.
- in use were Openvswitch-dpdk 2.5 and DPDK 2.2; Retest with DPDK 16.04 and
Openvswitch master is planned.

I will go on debugging this and let you know, but I wanted to give a heads
up to everyone.
In case this is a known issue for some of you please let me know.

Kind Regards,
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

P.S. I think it is a dpdk issue, but adding Daniele on CC to represent
ovs-dpdk as well.