From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f176.google.com (mail-yw0-f176.google.com [209.85.161.176]) by dpdk.org (Postfix) with ESMTP id 4EE15378E for ; Thu, 29 Sep 2016 08:58:37 +0200 (CEST) Received: by mail-yw0-f176.google.com with SMTP id i129so42418621ywe.2 for ; Wed, 28 Sep 2016 23:58:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical-com.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to:cc; bh=cf7JHhhwRQHeLgUm3Eas3otI7r+MK91ZKXL8T65B+mg=; b=O97o1xwK0ae35WPVpMeUiUIadJT8ak6ZJUr+HV7IN48MBcpAXisVtBaG7CGrINU90j Zg9PzksoANiT8DFPHIVGsbRH1ezHNA74cIehrQod7O6D7NygcVU5fn+6Lby68k+sXRsr pLX38gFucrbwmkc6A7F4Jftc/1uZOM6rBahCjLSLqNd9tkusT3tgfkxUzVTMsL+ghSGM px3jsFQh3EitiRgv4lBfwqhO8ICyzN5gtLCxM9k3RGVMu2FrEXr6YcpuPGMXPI8+04Lu EYLzIvJuDPTHk9Kl6+oDS5vfvgyHTEo9Es0FGFeFZBgXW5AoBrlVHBVwMGGqR0Y2tSck usWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=cf7JHhhwRQHeLgUm3Eas3otI7r+MK91ZKXL8T65B+mg=; b=EMpEQ8FZ2OqagrPMzc3rWZtwd7STY58Bnupz8XH69TbXyX0rkzL4cF2oS3uts4MOKX o4ylUBjOckTlkPARqC9252jjZg8fuSxl8apK8l6VZRbVLnmgeEdLpv9IjEhe+Kk9mGur a1+wwhFVSyD6//c+BHKleJWo0u3CKHG2evIZBMFGrAHZRKCNvwRIGdAheAFieMluXjZg TuFr+hg3fHMvhiRQvGGLK/XUeY+nra/6GdERRaa+y78gcMQakaa3wM/FjAxMi/C1d+2t bCDntSQxd3PdRAADJ4c+plOAbRqs9wFrX8f0SjdrhRuXWcOLJf+Hk7G4iPvRJYzMk+Pl E0rQ== X-Gm-Message-State: AE9vXwPGvCw68asINGMf7ijnU+K55cRMdxydS15MLO8sxhK+GrnyYtHCReD9P97QiaOq0XJ7HMEGDcHV6y0h60kn X-Received: by 10.129.51.202 with SMTP id z193mr26292375ywz.152.1475132316615; Wed, 28 Sep 2016 23:58:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.119.196 with HTTP; Wed, 28 Sep 2016 23:58:06 -0700 (PDT) From: Christian Ehrhardt Date: Thu, 29 Sep 2016 08:58:06 +0200 Message-ID: To: Daniele Di Proietto , James Page , dev Cc: Luca Boccassi Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] Did we reduce unnecessary linkage too well? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2016 06:58:37 -0000 Hi, I was finally getting to more deeply re-validate Openvswitch 2.6 together with DPDK 16.07. And I think I found a whiplash of our effort to reduce unnecessary hard linkage. Trying to avoid cross-posting, picking DPDK list and the main involved people on TO/CC. TL;DR: - pmd drivers are no more "auto"-loaded - adding -d ...so to all consuming applications feels obnoxious - do we really have to intentionally overlink some? - I hope I just overlook something trivial to fix this. Detail: Essentially in the past openvswitch was linked to "the one" combined library being libdpdk.so. Due to all the great work on separating the libraries for versioning and reducing overlinking I now have an openvswitch that only depends on the core dpdk libs. $ldd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk | grep rte librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1 librte_pmd_ring.so.2 => /usr/lib/x86_64-linux-gnu/librte_pmd_ring.so.2 librte_mbuf.so.2 => /usr/lib/x86_64-linux-gnu/librte_mbuf.so.2 librte_vhost.so.3 => /usr/lib/x86_64-linux-gnu/librte_vhost.so.3 librte_mempool.so.2 => /usr/lib/x86_64-linux-gnu/librte_mempool.so.2 librte_meter.so.1 => /usr/lib/x86_64-linux-gnu/librte_meter.so.1 librte_ring.so.1 => /usr/lib/x86_64-linux-gnu/librte_ring.so.1 librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2 librte_kvargs.so.1 => /usr/lib/x86_64-linux-gnu/librte_kvargs.so.1 As we all know that is due to the build of openvswitch now utilizing the linker script which is $cat /usr/lib/x86_64-linux-gnu/libdpdk.so GROUP ( librte_power.so librte_pmd_null_crypto.so librte_ip_frag.so librte_pmd_e1000.so librte_acl.so librte_port.so librte_mbuf.so librte_pmd_virtio.so librte_kvargs.so li brte_cryptodev.so librte_reorder.so librte_pmd_ring.so librte_eal.so librte_distributor.so librte_pmd_pcap.so librte_pmd_af_packet.so librte_pipeline.so librte_pdump.so lib rte_pmd_vhost.so librte_jobstats.so librte_pmd_enic.so librte_mempool.so librte_pmd_ixgbe.so librte_cmdline.so librte_meter.so librte_timer.so librte_kni.so librte_pmd_vmxn et3_uio.so librte_ring.so librte_pmd_fm10k.so librte_cfgfile.so librte_pmd_xenvirt.so librte_pmd_i40e.so librte_pmd_null.so librte_lpm.so librte_pmd_ena.so librte_vhost.so libethdev.so librte_pmd_bnxt.so librte_sched.so librte_table.so librte_pmd_bond.so librte_hash.so librte_pmd_cxgbe.so ) You see it lists the PMDs, but since this is more a plugin structure there are no references by openvswitch to them. So they don't make it into the eventual binary. Which is kind of what we wanted - avoid over-linking right? But this leads to IMHO unexpected behaviour. For example I have a system properly prepared with two ixgbe cards and start OVS-DPDK in "the right way" I get the issue of not having a valid PMD driver. "could not open network device dpdk0" => https://launchpadlibrarian.net/287083118/ovs-vswitchd-bad.log Now this can work if one tells EAL to load the proper driver, so after $ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=[...] -d /usr/lib/x86_64-linux-gnu/librte_pmd_ixgbe.so" Everything works fine now and as-expected. => https://launchpadlibrarian.net/287083132/ovs-vswitchd-good.log One can see ixgbe pmd driver loading and taking over the devices. I think this puts too much pressure to know the right drivers to the admin - so we can't really make this the way it should be right? It feels obnoxious having to add a full path to a .so file to a ovsdb parameter. It might be worth to note that I can reproduce the same behavior with l2fwd, without the -d to the lib on the same systemit quits with: EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye I considered this a test/developer tool where it is kind of ok. But now I think it is time to discuss it in the scope of more customer-centric dpdk consuming applications like openvswitch+dpdk. The build logs of how the tested openvswitch and dpdk were build are here - just in case one of you want/can spot an issue causing all this in there: - openvswitch (all of it, not just openvswitch-dpdk): https://launchpadlibrarian.net/284431032/buildlog_ubuntu-yakkety-amd64.openvswitch_2.6.0~git20160912.dc61b4e-0ubuntu4_BUILDING.txt.gz - dpdk-16.07: https://launchpadlibrarian.net/285321110/buildlog_ubuntu-yakkety-amd64.dpdk_16.07-0ubuntu3_BUILDING.txt.gz I haven't seen nor found a discussion on this - nor people testing my packages complainning so far -, but find it so weird that I start to think it is just me or my broken setup :-) So I guess there is hope that there is something obvious I overlook - better that than needing major reworks. Like a default search dir for the libs that I mismatch in my setup or anything like it? If not, should we do something - I hate to say that - to intentionally overlink dpdk libraries to get all pre-provided PMDs autoloaded? Kind Regards, -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd