[dpdk-dev] Questions on DPDK pkg-config

[dpdk-dev] Questions on DPDK pkg-config

[Harris, James R]

Hi,

SPDK would like to use DPDK’s pkg-config files rather than rolling our own linker arguments.  I’m running into a couple of issues – one looks like a bug, the other is not as clear.

First is the use of --as-needed (https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be0c8504fad).  We have run into problems with this in the past, specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS constructor functions.  --as-needed results in similar behavior to omitting --whole-archive when using static libraries – meaning the constructor function doesn’t get called and we cannot create any mempools since there are no registered mempool_ops.  I think this would also affect other uses of constructor functions within DPDK when using pkg-config, but this rte_mempool_ring one is the only that would impact SPDK so it’s the only one I’ve checked.

Second is that rte_bus_pci is not included in the pkg-config output.  SPDK relies on rte_bus_pci since we have our own DPDK drivers for things like nvme, virtio-blk and virtio-scsi and need access to rte_pci_read_config, rte_pci_write_config and rte_pci_register.  Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?

Any help would be appreciated.

Thanks,

-Jim

Re: [dpdk-dev] Questions on DPDK pkg-config

[Bruce Richardson]

On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
>    Hi,
> 
> 
>    SPDK would like to use DPDK’s pkg-config files rather than rolling our
>    own linker arguments.  I’m running into a couple of issues – one looks
>    like a bug, the other is not as clear.
> 
> 
>    First is the use of --as-needed
>    ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
>    0c8504fad).  We have run into problems with this in the past,
>    specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
>    constructor functions.  --as-needed results in similar behavior to
>    omitting --whole-archive when using static libraries – meaning the
>    constructor function doesn’t get called and we cannot create any
>    mempools since there are no registered mempool_ops.  I think this would
>    also affect other uses of constructor functions within DPDK when using
>    pkg-config, but this rte_mempool_ring one is the only that would impact
>    SPDK so it’s the only one I’ve checked.
> 
> 
>    Second is that rte_bus_pci is not included in the pkg-config output.
>    SPDK relies on rte_bus_pci since we have our own DPDK drivers for
>    things like nvme, virtio-blk and virtio-scsi and need access to
>    rte_pci_read_config, rte_pci_write_config and rte_pci_register.
>    Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
> 
> 
>    Any help would be appreciated.
> 
> 
>    Thanks,
> 
> 
>    -Jim
> 
Hi Jim,

what command are you using to get the libs etc. from pkg-config? For static
linking you need to pass the --static flag which will include all the libs
and drivers, including the pci bus driver. [See output below from my system after
running "ninja install"]. The --as-needed is required to prevent the shared
libs also being linked into a static build in this case.

/Bruce

$ PKG_CONFIG_PATH=/usr/local/lib/x86_64-linux-gnu/pkgconfig/ pkg-config --libs --static libdpdk
-L/usr/local/lib/x86_64-linux-gnu -Wl,--whole-archive -l:librte_common_cpt.a -l:librte_common_dpaax.a -l:librte_common_iavf.a -l:librte_common_octeontx.a -l:librte_common_octeontx2.a -l:librte_bus_dpaa.a -l:librte_bus_fslmc.a -l:librte_bus_ifpga.a -l:librte_bus_pci.a -l:librte_bus_vdev.a -l:librte_bus_vmbus.a -l:librte_mempool_bucket.a -l:librte_mempool_dpaa.a -l:librte_mempool_dpaa2.a -l:librte_mempool_octeontx.a -l:librte_mempool_octeontx2.a -l:librte_mempool_ring.a -l:librte_mempool_stack.a -l:librte_pmd_e1000.a -l:librte_pmd_ena.a -l:librte_pmd_enetc.a -l:librte_pmd_enic.a -l:librte_node.a -l:librte_graph.a -l:librte_bpf.a -l:librte_flow_classify.a -l:librte_pipeline.a -l:librte_table.a -l:librte_port.a -l:librte_fib.a -l:librte_ipsec.a -l:librte_vhost.a -l:librte_stack.a -l:librte_security.a -l:librte_sched.a -l:librte_reorder.a -l:librte_rib.a -l:librte_regexdev.a -l:librte_rawdev.a -l:librte_pdump.a -l:librte_power.a -l:librte_member.a -l:librte_lpm.a -l:librte_latencystats.a -l:librte_kni.a -l:librte_jobstats.a -l:librte_ip_frag.a -l:librte_gso.a -l:librte_gro.a -l:librte_eventdev.a -l:librte_efd.a -l:librte_distributor.a -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_cfgfile.a -l:librte_bitratestats.a -l:librte_bbdev.a -l:librte_acl.a -l:librte_timer.a -l:librte_hash.a -l:librte_metrics.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a -l:librte_kvargs.a -Wl,--no-whole-archive -lpcap -ljansson -Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -pthread -lm -ldl -lfdt -lpcap

Re: [dpdk-dev] Questions on DPDK pkg-config

[Harris, James R]

On 9/24/20, 2:27 AM, "Bruce Richardson" <bruce.richardson@intel.com> wrote:

    On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
    >    Hi,
    > 
    > 
    >    SPDK would like to use DPDK’s pkg-config files rather than rolling our
    >    own linker arguments.  I’m running into a couple of issues – one looks
    >    like a bug, the other is not as clear.
    > 
    > 
    >    First is the use of --as-needed
    >    ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
    >    0c8504fad).  We have run into problems with this in the past,
    >    specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
    >    constructor functions.  --as-needed results in similar behavior to
    >    omitting --whole-archive when using static libraries – meaning the
    >    constructor function doesn’t get called and we cannot create any
    >    mempools since there are no registered mempool_ops.  I think this would
    >    also affect other uses of constructor functions within DPDK when using
    >    pkg-config, but this rte_mempool_ring one is the only that would impact
    >    SPDK so it’s the only one I’ve checked.
    > 
    > 
    >    Second is that rte_bus_pci is not included in the pkg-config output.
    >    SPDK relies on rte_bus_pci since we have our own DPDK drivers for
    >    things like nvme, virtio-blk and virtio-scsi and need access to
    >    rte_pci_read_config, rte_pci_write_config and rte_pci_register.
    >    Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
    > 
    > 
    >    Any help would be appreciated.
    > 
    > 
    >    Thanks,
    > 
    > 
    >    -Jim
    > 
    Hi Jim,

    what command are you using to get the libs etc. from pkg-config? For static
    linking you need to pass the --static flag which will include all the libs
    and drivers, including the pci bus driver. [See output below from my system after
    running "ninja install"]. The --as-needed is required to prevent the shared
    libs also being linked into a static build in this case.

    /Bruce

<snip>

Yes, the output from pkg-config works fine for the static library use case.  I also see librte_bus_pci.a get emitted when specifying --static.

I chatted with Bruce offline about this, and we (mostly Bruce) root caused the issue with the shared library use case.  SPDK is using a non-standard DESTDIR and then not specifying the PMD directory via the -d command line argument.  So this needs to be fixed on the SPDK side.

Thanks,

Jim

Re: [dpdk-dev] Questions on DPDK pkg-config

[Bruce Richardson]

On Thu, Sep 24, 2020 at 03:34:12PM +0100, Harris, James R wrote:
> 
> 
> On 9/24/20, 2:27 AM, "Bruce Richardson" <bruce.richardson@intel.com> wrote:
> 
>     On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
>     >    Hi,
>     >
>     >
>     >    SPDK would like to use DPDK’s pkg-config files rather than rolling our
>     >    own linker arguments.  I’m running into a couple of issues – one looks
>     >    like a bug, the other is not as clear.
>     >
>     >
>     >    First is the use of --as-needed
>     >    ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
>     >    0c8504fad).  We have run into problems with this in the past,
>     >    specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
>     >    constructor functions.  --as-needed results in similar behavior to
>     >    omitting --whole-archive when using static libraries – meaning the
>     >    constructor function doesn’t get called and we cannot create any
>     >    mempools since there are no registered mempool_ops.  I think this would
>     >    also affect other uses of constructor functions within DPDK when using
>     >    pkg-config, but this rte_mempool_ring one is the only that would impact
>     >    SPDK so it’s the only one I’ve checked.
>     >
>     >
>     >    Second is that rte_bus_pci is not included in the pkg-config output.
>     >    SPDK relies on rte_bus_pci since we have our own DPDK drivers for
>     >    things like nvme, virtio-blk and virtio-scsi and need access to
>     >    rte_pci_read_config, rte_pci_write_config and rte_pci_register.
>     >    Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
>     >
>     >
>     >    Any help would be appreciated.
>     >
>     >
>     >    Thanks,
>     >
>     >
>     >    -Jim
>     >
>     Hi Jim,
> 
>     what command are you using to get the libs etc. from pkg-config? For static
>     linking you need to pass the --static flag which will include all the libs
>     and drivers, including the pci bus driver. [See output below from my system after
>     running "ninja install"]. The --as-needed is required to prevent the shared
>     libs also being linked into a static build in this case.
> 
>     /Bruce
> 
> <snip>
> 
> Yes, the output from pkg-config works fine for the static library use case.  I also see librte_bus_pci.a get emitted when specifying --static.
> 
> I chatted with Bruce offline about this, and we (mostly Bruce) root caused the issue with the shared library use case.  SPDK is using a non-standard DESTDIR and then not specifying the PMD directory via the -d command line argument.  So this needs to be fixed on the SPDK side.
> 
> Thanks,
> 
> Jim

As general follow-up information, if you are planning to install DPDK to a
non-system location on "ninja install" and run things from that location,
it's probably better to set the DPDK "prefix" option to the location rather
than using DESTDIR. 

Setting the prefix using "meson -Dprefix=" e.g. "meson
-Dprefix=$HOME/.local" will make the final path visible to the build so
should allow DPDK's default driver load path to be set correctly. On the
other hand, with DESTDIR, the value is only used at install time and does
not affect the build, meaning that EAL has no way of knowing the final
installation path.  [AFAIK: DESTDIR is designed for tasks like packaging
where you want to define a separate root folder for your temporary
installation]

Regards,
/Bruce