* [dpdk-dev] Questions on DPDK pkg-config
@ 2020-09-23 18:21 Harris, James R
2020-09-24 9:26 ` Bruce Richardson
0 siblings, 1 reply; 4+ messages in thread
From: Harris, James R @ 2020-09-23 18:21 UTC (permalink / raw)
To: dev, Richardson, Bruce
Hi,
SPDK would like to use DPDK’s pkg-config files rather than rolling our own linker arguments. I’m running into a couple of issues – one looks like a bug, the other is not as clear.
First is the use of --as-needed (https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be0c8504fad). We have run into problems with this in the past, specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS constructor functions. --as-needed results in similar behavior to omitting --whole-archive when using static libraries – meaning the constructor function doesn’t get called and we cannot create any mempools since there are no registered mempool_ops. I think this would also affect other uses of constructor functions within DPDK when using pkg-config, but this rte_mempool_ring one is the only that would impact SPDK so it’s the only one I’ve checked.
Second is that rte_bus_pci is not included in the pkg-config output. SPDK relies on rte_bus_pci since we have our own DPDK drivers for things like nvme, virtio-blk and virtio-scsi and need access to rte_pci_read_config, rte_pci_write_config and rte_pci_register. Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
Any help would be appreciated.
Thanks,
-Jim
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] Questions on DPDK pkg-config
2020-09-23 18:21 [dpdk-dev] Questions on DPDK pkg-config Harris, James R
@ 2020-09-24 9:26 ` Bruce Richardson
2020-09-24 14:34 ` Harris, James R
0 siblings, 1 reply; 4+ messages in thread
From: Bruce Richardson @ 2020-09-24 9:26 UTC (permalink / raw)
To: Harris, James R; +Cc: dev
On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
> Hi,
>
>
> SPDK would like to use DPDK’s pkg-config files rather than rolling our
> own linker arguments. I’m running into a couple of issues – one looks
> like a bug, the other is not as clear.
>
>
> First is the use of --as-needed
> ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
> 0c8504fad). We have run into problems with this in the past,
> specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
> constructor functions. --as-needed results in similar behavior to
> omitting --whole-archive when using static libraries – meaning the
> constructor function doesn’t get called and we cannot create any
> mempools since there are no registered mempool_ops. I think this would
> also affect other uses of constructor functions within DPDK when using
> pkg-config, but this rte_mempool_ring one is the only that would impact
> SPDK so it’s the only one I’ve checked.
>
>
> Second is that rte_bus_pci is not included in the pkg-config output.
> SPDK relies on rte_bus_pci since we have our own DPDK drivers for
> things like nvme, virtio-blk and virtio-scsi and need access to
> rte_pci_read_config, rte_pci_write_config and rte_pci_register.
> Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
>
>
> Any help would be appreciated.
>
>
> Thanks,
>
>
> -Jim
>
Hi Jim,
what command are you using to get the libs etc. from pkg-config? For static
linking you need to pass the --static flag which will include all the libs
and drivers, including the pci bus driver. [See output below from my system after
running "ninja install"]. The --as-needed is required to prevent the shared
libs also being linked into a static build in this case.
/Bruce
$ PKG_CONFIG_PATH=/usr/local/lib/x86_64-linux-gnu/pkgconfig/ pkg-config --libs --static libdpdk
-L/usr/local/lib/x86_64-linux-gnu -Wl,--whole-archive -l:librte_common_cpt.a -l:librte_common_dpaax.a -l:librte_common_iavf.a -l:librte_common_octeontx.a -l:librte_common_octeontx2.a -l:librte_bus_dpaa.a -l:librte_bus_fslmc.a -l:librte_bus_ifpga.a -l:librte_bus_pci.a -l:librte_bus_vdev.a -l:librte_bus_vmbus.a -l:librte_mempool_bucket.a -l:librte_mempool_dpaa.a -l:librte_mempool_dpaa2.a -l:librte_mempool_octeontx.a -l:librte_mempool_octeontx2.a -l:librte_mempool_ring.a -l:librte_mempool_stack.a -l:librte_pmd_e1000.a -l:librte_pmd_ena.a -l:librte_pmd_enetc.a -l:librte_pmd_enic.a -l:librte_node.a -l:librte_graph.a -l:librte_bpf.a -l:librte_flow_classify.a -l:librte_pipeline.a -l:librte_table.a -l:librte_port.a -l:librte_fib.a -l:librte_ipsec.a -l:librte_vhost.a -l:librte_stack.a -l:librte_security.a -l:librte_sched.a -l:librte_reorder.a -l:librte_rib.a -l:librte_regexdev.a -l:librte_rawdev.a -l:librte_pdump.a -l:librte_power.a -l:librte_member.a -l:librte_lpm.a -l:librte_latencystats.a -l:librte_kni.a -l:librte_jobstats.a -l:librte_ip_frag.a -l:librte_gso.a -l:librte_gro.a -l:librte_eventdev.a -l:librte_efd.a -l:librte_distributor.a -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_cfgfile.a -l:librte_bitratestats.a -l:librte_bbdev.a -l:librte_acl.a -l:librte_timer.a -l:librte_hash.a -l:librte_metrics.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a -l:librte_kvargs.a -Wl,--no-whole-archive -lpcap -ljansson -Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -pthread -lm -ldl -lfdt -lpcap
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] Questions on DPDK pkg-config
2020-09-24 9:26 ` Bruce Richardson
@ 2020-09-24 14:34 ` Harris, James R
2020-09-24 14:43 ` Bruce Richardson
0 siblings, 1 reply; 4+ messages in thread
From: Harris, James R @ 2020-09-24 14:34 UTC (permalink / raw)
To: Richardson, Bruce; +Cc: dev
On 9/24/20, 2:27 AM, "Bruce Richardson" <bruce.richardson@intel.com> wrote:
On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
> Hi,
>
>
> SPDK would like to use DPDK’s pkg-config files rather than rolling our
> own linker arguments. I’m running into a couple of issues – one looks
> like a bug, the other is not as clear.
>
>
> First is the use of --as-needed
> ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
> 0c8504fad). We have run into problems with this in the past,
> specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
> constructor functions. --as-needed results in similar behavior to
> omitting --whole-archive when using static libraries – meaning the
> constructor function doesn’t get called and we cannot create any
> mempools since there are no registered mempool_ops. I think this would
> also affect other uses of constructor functions within DPDK when using
> pkg-config, but this rte_mempool_ring one is the only that would impact
> SPDK so it’s the only one I’ve checked.
>
>
> Second is that rte_bus_pci is not included in the pkg-config output.
> SPDK relies on rte_bus_pci since we have our own DPDK drivers for
> things like nvme, virtio-blk and virtio-scsi and need access to
> rte_pci_read_config, rte_pci_write_config and rte_pci_register.
> Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
>
>
> Any help would be appreciated.
>
>
> Thanks,
>
>
> -Jim
>
Hi Jim,
what command are you using to get the libs etc. from pkg-config? For static
linking you need to pass the --static flag which will include all the libs
and drivers, including the pci bus driver. [See output below from my system after
running "ninja install"]. The --as-needed is required to prevent the shared
libs also being linked into a static build in this case.
/Bruce
<snip>
Yes, the output from pkg-config works fine for the static library use case. I also see librte_bus_pci.a get emitted when specifying --static.
I chatted with Bruce offline about this, and we (mostly Bruce) root caused the issue with the shared library use case. SPDK is using a non-standard DESTDIR and then not specifying the PMD directory via the -d command line argument. So this needs to be fixed on the SPDK side.
Thanks,
Jim
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [dpdk-dev] Questions on DPDK pkg-config
2020-09-24 14:34 ` Harris, James R
@ 2020-09-24 14:43 ` Bruce Richardson
0 siblings, 0 replies; 4+ messages in thread
From: Bruce Richardson @ 2020-09-24 14:43 UTC (permalink / raw)
To: Harris, James R; +Cc: dev
On Thu, Sep 24, 2020 at 03:34:12PM +0100, Harris, James R wrote:
>
>
> On 9/24/20, 2:27 AM, "Bruce Richardson" <bruce.richardson@intel.com> wrote:
>
> On Wed, Sep 23, 2020 at 07:21:16PM +0100, Harris, James R wrote:
> > Hi,
> >
> >
> > SPDK would like to use DPDK’s pkg-config files rather than rolling our
> > own linker arguments. I’m running into a couple of issues – one looks
> > like a bug, the other is not as clear.
> >
> >
> > First is the use of --as-needed
> > ([1]https://github.com/DPDK/dpdk/commit/b98447077b0609750c10b84b7b2e7be
> > 0c8504fad). We have run into problems with this in the past,
> > specifically related to the rte_mempool_ring MEMPOOL_REGISTER_OPS
> > constructor functions. --as-needed results in similar behavior to
> > omitting --whole-archive when using static libraries – meaning the
> > constructor function doesn’t get called and we cannot create any
> > mempools since there are no registered mempool_ops. I think this would
> > also affect other uses of constructor functions within DPDK when using
> > pkg-config, but this rte_mempool_ring one is the only that would impact
> > SPDK so it’s the only one I’ve checked.
> >
> >
> > Second is that rte_bus_pci is not included in the pkg-config output.
> > SPDK relies on rte_bus_pci since we have our own DPDK drivers for
> > things like nvme, virtio-blk and virtio-scsi and need access to
> > rte_pci_read_config, rte_pci_write_config and rte_pci_register.
> > Perhaps we could add bus_pci to the dpdk_libraries in lib/meson.build?
> >
> >
> > Any help would be appreciated.
> >
> >
> > Thanks,
> >
> >
> > -Jim
> >
> Hi Jim,
>
> what command are you using to get the libs etc. from pkg-config? For static
> linking you need to pass the --static flag which will include all the libs
> and drivers, including the pci bus driver. [See output below from my system after
> running "ninja install"]. The --as-needed is required to prevent the shared
> libs also being linked into a static build in this case.
>
> /Bruce
>
> <snip>
>
> Yes, the output from pkg-config works fine for the static library use case. I also see librte_bus_pci.a get emitted when specifying --static.
>
> I chatted with Bruce offline about this, and we (mostly Bruce) root caused the issue with the shared library use case. SPDK is using a non-standard DESTDIR and then not specifying the PMD directory via the -d command line argument. So this needs to be fixed on the SPDK side.
>
> Thanks,
>
> Jim
As general follow-up information, if you are planning to install DPDK to a
non-system location on "ninja install" and run things from that location,
it's probably better to set the DPDK "prefix" option to the location rather
than using DESTDIR.
Setting the prefix using "meson -Dprefix=" e.g. "meson
-Dprefix=$HOME/.local" will make the final path visible to the build so
should allow DPDK's default driver load path to be set correctly. On the
other hand, with DESTDIR, the value is only used at install time and does
not affect the build, meaning that EAL has no way of knowing the final
installation path. [AFAIK: DESTDIR is designed for tasks like packaging
where you want to define a separate root folder for your temporary
installation]
Regards,
/Bruce
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-09-24 14:43 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-23 18:21 [dpdk-dev] Questions on DPDK pkg-config Harris, James R
2020-09-24 9:26 ` Bruce Richardson
2020-09-24 14:34 ` Harris, James R
2020-09-24 14:43 ` Bruce Richardson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).