From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5B2D0A0613 for ; Thu, 29 Aug 2019 17:25:43 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6AB6F1D157; Thu, 29 Aug 2019 17:25:42 +0200 (CEST) Received: from youngberry.canonical.com (youngberry.canonical.com [91.189.89.112]) by dpdk.org (Postfix) with ESMTP id 9F52E1C1F1 for ; Thu, 29 Aug 2019 17:25:40 +0200 (CEST) Received: from mail-vk1-f199.google.com ([209.85.221.199]) by youngberry.canonical.com with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1i3MIu-0002oT-8i for dev@dpdk.org; Thu, 29 Aug 2019 15:25:40 +0000 Received: by mail-vk1-f199.google.com with SMTP id s17so1445274vkm.10 for ; Thu, 29 Aug 2019 08:25:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Ywk0BahBOvFf9duv48jgQ0VVR7vHRmgj99qBny1H7Eo=; b=PqrOylEnhNhy1kAj9YSQ+MfDxuw4xEQvVGR/ipIEh+V8o5fAg0ytP2f3mNaL4/N0et wXNcbcdlPdyByt/SdRezIYq4VGcUecxQJqRyUScJ0IrBKde+yMon2nzyli9Sxxbs3CiF R0zUzGOcKKTH3jT3rbt5Zbcx/TdmiWfniYwjBYGXTXlAMbOlvuZJpcnqLfYzX561KI/F uwZoxiErvnfZS6aTzDYIwLXzYfxOXFDUwnU/ohcUvFoQDt7Y8zrEcNkSVwgkR6w82Coi VE+I/J4OenZn9eAPPKK/muIYWPn3sSM0qBXahJybAPyckX2k9y3DHMCYfaHkR+IdTCBL kXpQ== X-Gm-Message-State: APjAAAVfHgMIBmuv+h+Nc2IXupHEhufEZ03HVspzCGvxqS1FHWsoWAAu 7iTpiTPXIQhleTBCAnOwLwcnYbLvtnIewqhJuqI41QMrwaL66NqoKHkTfo6JXmPK/EJSZ8/jVJu RkT2YvtNHCDbcq0R4bOjToPko+5I+f3P5e20j X-Received: by 2002:ab0:308c:: with SMTP id h12mr4873952ual.76.1567092339283; Thu, 29 Aug 2019 08:25:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqwJ9N55Zmqhla5McpHqxmACPfIQtHJCghVtbPiBRtSQAsTuapRYNmzceU5NNp/p2GQTtW+GoLXVa5KlxQPtPpU= X-Received: by 2002:ab0:308c:: with SMTP id h12mr4873935ual.76.1567092338911; Thu, 29 Aug 2019 08:25:38 -0700 (PDT) MIME-Version: 1.0 References: <20190828122752.27887-1-christian.ehrhardt@canonical.com> In-Reply-To: From: Christian Ehrhardt Date: Thu, 29 Aug 2019 17:25:12 +0200 Message-ID: To: Aaron Conole Cc: dev , Kevin Laatz , Luca Boccassi , Thomas Monjalon Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH] build: avoid --as-needed as it causes overlinking X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Aug 29, 2019 at 12:18 PM Christian Ehrhardt wrote: > > On Wed, Aug 28, 2019 at 5:23 PM Aaron Conole wrote: > > > > Aaron Conole writes: > > > > > Christian Ehrhardt writes: > > > > > >> On Wed, Aug 28, 2019 at 3:53 PM Aaron Conole wrote: > > >>> > > >>> Christian Ehrhardt writes: > > >>> > > >>> > A while ago telemetry was added in 57ae0ec6 and it also added as-needed > > >>> > to config/meson.build. This seems no more needed these days as due to other > > >>> > build changes the ordering in buildlogs is: > > >>> > [...] -lrte_telemetry [...] -Wl,--no-as-needed [...] > > >>> > Which means telemetry no more benefits from --no-as-needed anyway. > > >>> > > > >>> > Overlinking problems get triggered by the meson generated pkgconfig which > > >>> > will have: > > >>> > [...] -Wl,--no-as-needed > > >>> > This will overlink and in addition anything that follows > > >>> > as it also doesn't wrap back to --as-needed. So if a projects includes > > >>> > dpdk libs + it will also consider with --no-as-needed. > > >>> > > > >>> > Fixes: https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1841759 > > >>> > > > >>> > Signed-off-by: Christian Ehrhardt > > >>> > --- > > >>> > > >>> Hi Christian, > > >>> > > >>> I agree this is something to be fixed. It will need additional work, > > >>> though: > > >>> > > >>> https://travis-ci.com/ovsrobot/dpdk/builds/124909245 > > >>> > > >> > > >> Thanks for the Link Aaron, yet I'm puzzled what to do there atm. > > >> > > >> The kind of error I found in the failing logs were misleading at first: > > >> - linker can't find -lvirt / -lpqos / ... > > >> well the test env needs to install them, maybe it was added as > > >> dependency by accident before? > > > > > > Not sure about this. It's strange to require that we *install* the > > > libraries before we can unit test them. After all, if I'm going to > > > potentially replace my previously installed libraries, I definitely want > > > to know that the unit tests are passing. > > > > > >> I'd understand (due to the change) if it would complain about missing symbols > > >> (no more added due to as-needed, but then for some reason needed) > > >> But this is vice versa, it just doesn't find the libs in the build env > > >> - error: unrecognized command line option '-Wformat-truncation' > > >> I don't see how I'd cause this ... > > >> => Maybe this is just an artifact that is even part of the normal/good tests? > > > > > > I don't think so - but there's a simple change. I've pushed to my own > > > branch and you can see the builds: > > > > > > https://travis-ci.org/orgcandman/dpdk/branches using the same > > > series_6154 branch name. > > > > > >> Comparing former logs - last good test was > > >> https://travis-ci.com/ovsrobot/dpdk/builds/124875383 > > >> This first seemed more helpful. > > >> > > >> DPDK:fast-tests / eal_flags_w_opt_autotest FAIL > > >> DPDK:fast-tests / func_reentrancy_autotest FAIL > > >> DPDK:fast-tests / mbuf_autotest FAIL > > >> DPDK:fast-tests / mempool_autotest FAIL > > >> DPDK:fast-tests / ring_pmd_autotest FAIL > > >> DPDK:fast-tests / sched_autotest FAIL > > >> DPDK:fast-tests / table_autotest FAIL > > >> [...] > > >> Overall about 14/60 of the tests failed with no recognizable pattern > > >> why just those and not the others. > > > > > > Good question :) > > > > > >> I only see "Full log written ... on_error", so I can't directly > > >> compare how a good run would look in the configure/build stage. > > >> Looking just at the bad case there are plenty of messages like > > >> - "no available hugepages" > > >> - "cannot reserve memory", .. > > >> But all those indicate more a flaky test(-env) than an error in the > > >> commit, there must be more to it. > > > > > > Okay. Fair enough. > > > > > >> @Aaron is there a good way to get the rest of the log for a good case > > >> to compare? > > > > > > Let's wait for https://travis-ci.org/orgcandman/dpdk/builds/577910388 to > > > spit out some details. > > > > Oops - forgot to push the revert: > > > > https://travis-ci.org/orgcandman/dpdk/builds/577918381 is the correct > > build. > > > > Sorry. > > Thanks Aaron, breaking down the actual different test results ... > > Tests: eal_flags_w_opt_autotest eal_flags_b_opt_autotest > EAL: failed to parse device "00FF:09:0B.3" > EAL: Unable to parse device '00FF:09:0B.3' > > Test: func_reentrancy_autotest > mempool create/lookup: common object allocated 0 times (should be 1) > > Tests: flow_classify_autotest ring_pmd_autotest sched_autotest > table_autotest bitratestats_autotest distributor_autotest > latencystats_autotest reorder_autotest > MBUF: error setting mempool handler > Cannot init mbuf pool on socket 0 > > Test: mbuf_autotest > MBUF: error setting mempool handler > cannot allocate mbuf pool > > Test: mempool_autotest > cannot allocate mp_nocache mempool > > Test: eventdev_common_autotest > Tests not executed > > > I built DPDK off of my commit locally and tried to run the tests, but > they work fine > > > # app/test/dpdk-test --no-huge -l 0-1 --pci-whitelist 00FF:09:0B.3 > EAL: Detected 4 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Static memory layout is selected, amount of reserved memory can > be adjusted with -m or --socket-mem > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: Probing VFIO support... > EAL: cannot open VFIO container, error 2 (No such file or directory) > EAL: VFIO support could not be initialized > APP: HPET is not enabled, using TSC as default timer > > Works just fine ?! > And if I really break it it fails as expected > > # app/test/dpdk-test --no-huge -l 0-1 --pci-whitelist 00FF:09:0B.3xxxx > EAL: Detected 4 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Static memory layout is selected, amount of reserved memory can > be adjusted with -m or --socket-mem > EAL: failed to parse device "00FF:09:0B.3xxxx" > EAL: Unable to parse device '00FF:09:0B.3xxxx' > > And trying another one of the cases that are most common "error > setting mempool handler" works fine as well > > # app/test/dpdk-test --no-huge -l 0-1 > EAL: Detected 4 lcore(s) > EAL: Detected 1 NUMA nodes > EAL: Static memory layout is selected, amount of reserved memory can > be adjusted with -m or --socket-mem > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket > EAL: Selected IOVA mode 'VA' > EAL: Probing VFIO support... > EAL: cannot open VFIO container, error 2 (No such file or directory) > EAL: VFIO support could not be initialized > EAL: PCI device 0000:00:1f.6 on NUMA socket -1 > EAL: Invalid NUMA socket, default to 0 > EAL: probe driver: 8086:15d8 net_e1000_em > APP: HPET is not enabled, using TSC as default timer > RTE>>flow_classify_autotest > Created table_acl for for IPv4 five tuple packets > Allocated mbuf pool on socket 0 > Set up IPv4 UDP traffic > ETH pktlen 14 > ETH + IPv4 pktlen 34 > ETH + IPv4 + UDP pktlen 42 > > Set up IPv4 TCP traffic > ETH pktlen 14 > ETH + IPv4 pktlen 34 > ETH + IPv4 + TCP pktlen 54 > > Set up IPv4 SCTP traffic > ETH pktlen 14 > ETH + IPv4 pktlen 34 > ETH + IPv4 + SCTP pktlen 42 > > Test OK > > > /me is still not seeing what would be wrong with it in this test environment :-/ > Especially since the change is on linking, that should be a do-or-die > breakage and not a (what seems random) subset of test fails. > > @Aaron, thanks for your help so far. Is there an easy way to run > exactly my code once again to see it if works this time? > Or was https://travis-ci.org/orgcandman/dpdk/builds/577910388 exactly > that already (and again failed at the same tests). > Thanks to Luca's support I set up travis for my github and experimented with some ideas we had. While I still don't understand how exactly we break those tests in particular, we have found a way to fix the generated pkg-config without breaking the tests: Bad: https://travis-ci.org/cpaelzer/dpdk/builds/578391327 Good: https://travis-ci.org/cpaelzer/dpdk/builds/578370028 I'll submit a v2 to this mail in a few minutes.