From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7724CA0524; Mon, 24 Feb 2020 09:48:24 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E528C1BFA5; Mon, 24 Feb 2020 09:48:23 +0100 (CET) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 208D91BE85 for ; Mon, 24 Feb 2020 09:48:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582534102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zoAOzwrM5gluQex0+mWYSRMPaSarPJVsmfBzZLBEdDw=; b=ezQXQllaswRObiNTaDFQXxYPY4wwxao3VDZboLL2I3kY2IMK7G9wjCDRpcIz9xBkej0H1L AT1WhQXJ2vQDKN9fEZZxR1xYOUnCpsbU3VIK724o9ZaiLmfzg8UFPbrfGkn0ZoJub82wQ/ LCG8xYhnYiPxpzqIe+2B0PIDR3Cyibw= Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-11-oiJ0Xc9qOnCncWTlzIex9Q-1; Mon, 24 Feb 2020 03:48:20 -0500 X-MC-Unique: oiJ0Xc9qOnCncWTlzIex9Q-1 Received: by mail-vk1-f198.google.com with SMTP id m25so4141135vko.19 for ; Mon, 24 Feb 2020 00:48:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=zoAOzwrM5gluQex0+mWYSRMPaSarPJVsmfBzZLBEdDw=; b=j4C8GRqs6o5KsnfyAavGf5ivJaTAmnrI90aA8FDJdbs34YWNYRctyce29XI4Bk/J2n lhzjRJ8sAswBGvdc3UFBXxNO+162Ca66oWZo4dbSzfBiJVJt54inddLBWawptUTL8oNR mO3nyHtpGqX6wBn9UCF7V/An3DEgTTb6/AWeHNA8+ZBMzFF/Pz6gMjljVYJmWjc2TcBT GCsJqU4oRoHzM++vqNFA+wW6iP1ZMy+mSQx2HfQyJjqJaRTq3Fy/4ZtlPNyGi+tW92TB hLYGEe9bUeLSk+5FDC2pWahP+WaD7jfQAXi5Aa++4fwnR2Bb2eRFxmnxlhZCOh+4Kk49 IaMQ== X-Gm-Message-State: APjAAAX/3LDVfcohu+KT43OwjzUicDmI5fDRET8Ra8YagEWsfbvmoi0N uc07AeMGsceQqKNY+LH8qY7U0Vy13TTLPOhAtjkbWzldDGiYTCQatYWatK5cTzAaKgPJw5R+wOa XkhIvImYKhgGuxhgMnrk= X-Received: by 2002:a67:f315:: with SMTP id p21mr26705573vsf.39.1582534099611; Mon, 24 Feb 2020 00:48:19 -0800 (PST) X-Google-Smtp-Source: APXvYqwcoY7s0RhAaxD1lrh6r1ot+5GxYSSJl/ISuh+lyVGaeicL/hj6M4CZ6VQS00RA0RNmoTJhkfsVWc3zjKY4kJQ= X-Received: by 2002:a67:f315:: with SMTP id p21mr26705552vsf.39.1582534099153; Mon, 24 Feb 2020 00:48:19 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: David Marchand Date: Mon, 24 Feb 2020 09:48:08 +0100 Message-ID: To: "humin (Q)" Cc: "dev@dpdk.org" , "Zhouchang (Forest)" , Xiedonghui , "liudongdong (C)" , lihuisong , "Huwei (Xavier)" , "Burakov, Anatoly" , Thomas Monjalon , Maxime Coquelin X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [RFC]app/testpmd: time-consuming question of mlockall function execution X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hello, On Mon, Feb 24, 2020 at 7:35 AM humin (Q) wrote: > We found that if OS transparent hugepage uses non 'always', mlockall func= tion in the main function of testpmd takes more than 25s to execute. > The results of running on both x86 and ARM are the same. It's very unreas= onable and deadly. The enable status of transparent hugepage on OS can be v= iewed by the following command. > [root@X6000-C23-1U dpdk]#cat /sys/kernel/mm/transparent_hugepage/enabled > always [madvise] never > > Transparent hugepage on ARM is configured as 'madvise', 'never' or 'alway= s', testpmd runs with using strace as follows=EF=BC=9A > ******************************* Transparent hugepage is configured as 'ma= dvise' ******************************* [root@X6000-C23-1U dpdk]# strace -T= -e trace=3Dmlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --iova-mode=3Dva -- -= i [snip] > Interactive-mode selected > mlockall(MCL_CURRENT|MCL_FUTURE) =3D 0 <25.736362> > <---------------------- Hang for 25 seconds [snip] > > ***************************** Transparent hugepage is configured as 'nev= er' ********************************* [root@X6000-C23-1U dpdk]# strace -T = -e trace=3Dmlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --iova-mode=3Dva -- -i [snip] > Interactive-mode selected > mlockall(MCL_CURRENT|MCL_FUTURE) =3D 0 <25.737757> > <---------------------- Hang for 25 seconds [snip] > > ***************************** Transparent hugepage is configured as 'alw= ays' ********************************* [root@X6000-C23-1U dpdk]# strace -T= -e trace=3Dmlockall testpmd -l 1-4 -w > 0000:7d:01.0 --iova-mode=3Dva -- -i > strace: Can't stat 'testpmd': No such file or directory [root@X6000-C23-1= U dpdk]# strace -T -e trace=3Dmlockall ./testpmd -l 1-4 -w 0000:7d:01.0 --i= ova-mode=3Dva -- -i [snip] > Interactive-mode selected > mlockall(MCL_CURRENT|MCL_FUTURE) =3D 0 <0.208571> > <---------------------- No Hang [snip] > > We have also seen some discussions on this issue in following page: > https://bugzilla.redhat.com/show_bug.cgi?id=3D1786923 > > David Marchand has a related patch, as following page: > https://github.com/david-marchand/dpdk/commit/f9e1b9fa101c9f4f16c0717401a= 55790aecc6484 > but this patch doesn't seem to have been submitted to the community. Yes, this is not ready, I worked on this locally since then (and the last version is not pushed to my github). The main problem is that I have not been able to reproduce Eelco issue so far which justified the addition of mlockall(). > Transparent hugepage on ARM is configured as 'madvise' or 'never', testpm= d runs with using strace as follows=EF=BC=9A > ******************************************************* > [root@X6000-C23-1U app]# strace -T -e trace=3Dmlockall ./testpmd -l 1-4 -= w > 0000:7d:01.0 --iova-mode=3Dva -- -i [snip] > Interactive-mode selected > mlockall(MCL_CURRENT|MCL_FUTURE|MCL_ONFAULT) =3D 0 <1.955947> > <---------------------- Hang for less than 2 seconds > testpmd: create a new mbuf pool : n=3D171456, size=3D= 2176, socket=3D0 > testpmd: preferred mempool ops selected: ring_mp_mc Done > testpmd> quit > > Bye... > +++ exited with 0 +++ > > We'd like to know what is the current development on this issue in dpdk c= ommunity. Thanks There is also a report about coredumps being huge. On this topic, there might be something to do with madvise + MADV_DONTDUMP. I see various options from the hardest to the easiest: - drop multiprocess support, - rework the memory allocator so that this kind of side effect (mapping a huge number of unused pages) is not hit, - change testpmd so that it locks only the pages of interest (this is the patch that I had been looking at for now), - change testpmd so that it does not call mlockall by default, The last option is likely the quickest workaround. I write "options", but the last two points feel like "band aid" fixes. And it does not solve the problem for other applications that will have to implement similar workarounds. Anatoly warned that touching the memory allocator is going to be hell. Quite sad to reach this state because it feels like people are just starting to hit the changes that entered dpdk 18.05. Of course, other ideas welcome! -- David Marchand