[Question/Issue] pkgconfig does not include dpdk/drivers libs

[Question/Issue] pkgconfig does not include dpdk/drivers libs

[Abhineet Pandey]

[-- Attachment #1: Type: text/plain, Size: 2177 bytes --]

I’m consuming dpdk via spdk.
I was trying to use pkg-config for a Makefile that I’m writing,

Executing:
PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs --static libdpdk

Output:
-Wl,--whole-archive -Wl,--no-whole-archive -Wl,--as-needed -pthread -L/home/nutanix/exact_spdk/spdk/dpdk/build/lib -l:librte_bus_pci.a -l:librte_bus_vdev.a -l:librte_mempool_ring.a -l:librte_vhost.a -l:librte_security.a -l:librte_reorder.a -l:librte_power.a -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_timer.a -l:librte_hash.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a -l:librte_kvargs.a -lrte_vhost -lrte_security -lrte_reorder -lrte_power -lrte_cryptodev -lrte_compressdev -lrte_timer -lrte_hash -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lm -ldl -lnuma

At a closer look, you’ll see -l:librte_bus_pci.a -l:librte_bus_vdev.a -l:librte_mempool_ring.a, but you won’t find corresponding -lrte_bus_pci, -lrte_bus_vdev, -lrte_mempool_ring. This showed up in my use case as rte_mempool_ring has some functions which execute on startup via __attribute__((constructor)), and they did not execute thus causing issues.

I think the issue is that in pkgconfig we reference ‘dpdk_libraries’, which is updated libs/meson.build, but not updated in drivers/meson.build.

If I apply this fix:

--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -277,7 +277,7 @@ foreach subpath:subdirs
         static_dep = declare_dependency(
                 include_directories: includes,
                 dependencies: static_deps)
-
+        dpdk_libraries = [shared_lib] + dpdk_libraries
          dpdk_drivers += static_lib

Then things work well and I see -lrte_bus_pci, -lrte_bus_vdev, -lrte_mempool_ring in output of:
PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs --static libdpdk

Please let me know if the current behavior is expected, or this change is needed.

Regards
Abhineet Pandey

[-- Attachment #2: Type: text/html, Size: 4327 bytes --]

Re: [Question/Issue] pkgconfig does not include dpdk/drivers libs

[Bruce Richardson]

On Thu, Aug 03, 2023 at 09:15:11AM +0000, Abhineet Pandey wrote:
>    I’m consuming dpdk via spdk.
>    I was trying to use pkg-config for a Makefile that I’m writing,
>    Executing:
> 
>    PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs --static
>    libdpdk
>    Output:
>    -Wl,--whole-archive -Wl,--no-whole-archive -Wl,--as-needed -pthread
>    -L/home/nutanix/exact_spdk/spdk/dpdk/build/lib -l:librte_bus_pci.a
>    -l:librte_bus_vdev.a -l:librte_mempool_ring.a -l:librte_vhost.a
>    -l:librte_security.a -l:librte_reorder.a -l:librte_power.a
>    -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_timer.a
>    -l:librte_hash.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a
>    -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a
>    -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a
>    -l:librte_kvargs.a -lrte_vhost -lrte_security -lrte_reorder -lrte_power
>    -lrte_cryptodev -lrte_compressdev -lrte_timer -lrte_hash -lrte_cmdline
>    -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool
>    -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lm -ldl
>    -lnuma
>    At a closer look, you’ll see -l:librte_bus_pci.a -l:librte_bus_vdev.a
>    -l:librte_mempool_ring.a, but you won’t find corresponding
>    -lrte_bus_pci, -lrte_bus_vdev, -lrte_mempool_ring. This showed up in my
>    use case as rte_mempool_ring has some functions which execute on
>    startup via __attribute__((constructor)), and they did not execute thus
>    causing issues.

I think there are two separate issues you are flagging here.
1. Absense of -lrte_bus_pci and similar flags
2. Missing constructor runs.

For the former, the behaviour is exactly as expected. You have specified
static linkage on the pkg-config commandline, so pkg-config is listing out
the libraries to link against only in static form. With
-l:librte_bus_pci.a, where is no need for -lrte_bus_pci since the pci bus
library is already linked in.

For the missing constructors, the issue is separate. That implies that when
the static libs are linked, the constructors are not getting included.
Normally, this is done by specifying the --whole-archive flag to the
linker, but in your case above, that is immediately followed by
--no-whole-archive which counteracts it! I suspect you may have a bug in
your pkg-config. By any chance are you using pkg-config 0.27? There is
known issue with it not linking static libs correctly, see note in [1]

/Bruce

[1] https://doc.dpdk.org/guides-23.07/linux_gsg/sys_reqs.html#compilation-of-the-dpdk

Re: [Question/Issue] pkgconfig does not include dpdk/drivers libs

[Abhineet Pandey]

[-- Attachment #1: Type: text/plain, Size: 4034 bytes --]

Hi Bruce,
Thanks for taking a look at this.

Talking about the issues you pointed,
1. Absense of -lrte_bus_pci and similar flags
Even if I don’t give –static flag, I don’t see -lrte_bus_pci (and  -lrte_bus_vdev, -lrte_mempool_ring).
Attaching output,
PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs libdpdk
-Wl,--as-needed -L/home/nutanix/spdk/dpdk/build/lib -lrte_vhost -lrte_security -lrte_reorder -lrte_dmadev -lrte_power -lrte_cryptodev -lrte_compressdev -lrte_timer -lrte_hash -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs
Not sure if this is expected.

2. Missing constructor runs.
Got your point. I’m using pkg-config  0.27.1. Will check how to avoid the automatic No Whole Archive thing. Still not sure about point 1 though.

Regards,
Abhineet

From: Bruce Richardson <bruce.richardson@intel.com>
Date: Thursday, 3 August 2023 at 3:46 PM
To: Abhineet Pandey <abhineet.pandey@nutanix.com>
Cc: dev@dpdk.org <dev@dpdk.org>
Subject: Re: [Question/Issue] pkgconfig does not include dpdk/drivers libs
On Thu, Aug 03, 2023 at 09:15:11AM +0000, Abhineet Pandey wrote:
>    I’m consuming dpdk via spdk.
>    I was trying to use pkg-config for a Makefile that I’m writing,
>    Executing:
>
>    PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs --static
>    libdpdk
>    Output:
>    -Wl,--whole-archive -Wl,--no-whole-archive -Wl,--as-needed -pthread
>    -L/home/nutanix/exact_spdk/spdk/dpdk/build/lib -l:librte_bus_pci.a
>    -l:librte_bus_vdev.a -l:librte_mempool_ring.a -l:librte_vhost.a
>    -l:librte_security.a -l:librte_reorder.a -l:librte_power.a
>    -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_timer.a
>    -l:librte_hash.a -l:librte_cmdline.a -l:librte_pci.a -l:librte_ethdev.a
>    -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a -l:librte_mempool.a
>    -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a -l:librte_telemetry.a
>    -l:librte_kvargs.a -lrte_vhost -lrte_security -lrte_reorder -lrte_power
>    -lrte_cryptodev -lrte_compressdev -lrte_timer -lrte_hash -lrte_cmdline
>    -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool
>    -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lm -ldl
>    -lnuma
>    At a closer look, you’ll see -l:librte_bus_pci.a -l:librte_bus_vdev.a
>    -l:librte_mempool_ring.a, but you won’t find corresponding
>    -lrte_bus_pci, -lrte_bus_vdev, -lrte_mempool_ring. This showed up in my
>    use case as rte_mempool_ring has some functions which execute on
>    startup via __attribute__((constructor)), and they did not execute thus
>    causing issues.

I think there are two separate issues you are flagging here.
1. Absense of -lrte_bus_pci and similar flags
2. Missing constructor runs.

For the former, the behaviour is exactly as expected. You have specified
static linkage on the pkg-config commandline, so pkg-config is listing out
the libraries to link against only in static form. With
-l:librte_bus_pci.a, where is no need for -lrte_bus_pci since the pci bus
library is already linked in.

For the missing constructors, the issue is separate. That implies that when
the static libs are linked, the constructors are not getting included.
Normally, this is done by specifying the --whole-archive flag to the
linker, but in your case above, that is immediately followed by
--no-whole-archive which counteracts it! I suspect you may have a bug in
your pkg-config. By any chance are you using pkg-config 0.27? There is
known issue with it not linking static libs correctly, see note in [1]

/Bruce

[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__doc.dpdk.org_guides-2D23.07_linux-5Fgsg_sys-5Freqs.html-23compilation-2Dof-2Dthe-2Ddpdk&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=UaVZAw32qzYwqV0iyjkICkaNhB8peZhhEJSs22LHVbI&m=NHXDGIxEa3NlLGmshhz7un6SMT4NlACKBYA39xVBj3VGki9GCytR-HAHnQi0cM4S&s=OSlZ0EizFQ8Xa1HkrPrAywHi_kuZHJHhTe48KKtL-AA&e=

[-- Attachment #2: Type: text/html, Size: 7371 bytes --]

Re: [Question/Issue] pkgconfig does not include dpdk/drivers libs

[Bruce Richardson]

On Thu, Aug 03, 2023 at 12:51:36PM +0000, Abhineet Pandey wrote:
>    Hi Bruce,
>    Thanks for taking a look at this.
>    Talking about the issues you pointed,
>    1. Absense of -lrte_bus_pci and similar flags
>    Even if I don’t give –static flag, I don’t see -lrte_bus_pci (and
>    -lrte_bus_vdev, -lrte_mempool_ring).
>    Attaching output,
>    PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs libdpdk
> 
>    -Wl,--as-needed -L/home/nutanix/spdk/dpdk/build/lib -lrte_vhost
>    -lrte_security -lrte_reorder -lrte_dmadev -lrte_power -lrte_cryptodev
>    -lrte_compressdev -lrte_timer -lrte_hash -lrte_cmdline -lrte_pci
>    -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu
>    -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs
>    Not sure if this is expected.

Yes, that is expected. For shared-library builds, the drivers are
dynamically loaded at runtime, rather than being linked directly to the
app. This allows drivers to be added/removed from an install without
breaking anything. To deploy DPDK to a platform, you only need copy over
the .so files you need for the hardware on that platform.

>    2. Missing constructor runs.
>    Got your point. I’m using pkg-config  0.27.1. Will check how to avoid
>    the automatic No Whole Archive thing. Still not sure about point 1
>    though.

I'd recommend installing pkgconf, or getting an updated pkg-config binary.

>    Regards,
>    Abhineet
> 
> 
>    From: Bruce Richardson <bruce.richardson@intel.com>
>    Date: Thursday, 3 August 2023 at 3:46 PM
>    To: Abhineet Pandey <abhineet.pandey@nutanix.com>
>    Cc: dev@dpdk.org <dev@dpdk.org>
>    Subject: Re: [Question/Issue] pkgconfig does not include dpdk/drivers
>    libs
> 
>    On Thu, Aug 03, 2023 at 09:15:11AM +0000, Abhineet Pandey wrote:
>    >    I’m consuming dpdk via spdk.
>    >    I was trying to use pkg-config for a Makefile that I’m writing,
>    >    Executing:
>    >
>    >    PKG_CONFIG_PATH=dpdk/build/lib/pkgconfig pkg-config --libs
>    --static
>    >    libdpdk
>    >    Output:
>    >    -Wl,--whole-archive -Wl,--no-whole-archive -Wl,--as-needed
>    -pthread
>    >    -L/home/nutanix/exact_spdk/spdk/dpdk/build/lib -l:librte_bus_pci.a
>    >    -l:librte_bus_vdev.a -l:librte_mempool_ring.a -l:librte_vhost.a
>    >    -l:librte_security.a -l:librte_reorder.a -l:librte_power.a
>    >    -l:librte_cryptodev.a -l:librte_compressdev.a -l:librte_timer.a
>    >    -l:librte_hash.a -l:librte_cmdline.a -l:librte_pci.a
>    -l:librte_ethdev.a
>    >    -l:librte_meter.a -l:librte_net.a -l:librte_mbuf.a
>    -l:librte_mempool.a
>    >    -l:librte_rcu.a -l:librte_ring.a -l:librte_eal.a
>    -l:librte_telemetry.a
>    >    -l:librte_kvargs.a -lrte_vhost -lrte_security -lrte_reorder
>    -lrte_power
>    >    -lrte_cryptodev -lrte_compressdev -lrte_timer -lrte_hash
>    -lrte_cmdline
>    >    -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf
>    -lrte_mempool
>    >    -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lm
>    -ldl
>    >    -lnuma
>    >    At a closer look, you’ll see -l:librte_bus_pci.a
>    -l:librte_bus_vdev.a
>    >    -l:librte_mempool_ring.a, but you won’t find corresponding
>    >    -lrte_bus_pci, -lrte_bus_vdev, -lrte_mempool_ring. This showed up
>    in my
>    >    use case as rte_mempool_ring has some functions which execute on
>    >    startup via __attribute__((constructor)), and they did not execute
>    thus
>    >    causing issues.
>    I think there are two separate issues you are flagging here.
>    1. Absense of -lrte_bus_pci and similar flags
>    2. Missing constructor runs.
>    For the former, the behaviour is exactly as expected. You have
>    specified
>    static linkage on the pkg-config commandline, so pkg-config is listing
>    out
>    the libraries to link against only in static form. With
>    -l:librte_bus_pci.a, where is no need for -lrte_bus_pci since the pci
>    bus
>    library is already linked in.
>    For the missing constructors, the issue is separate. That implies that
>    when
>    the static libs are linked, the constructors are not getting included.
>    Normally, this is done by specifying the --whole-archive flag to the
>    linker, but in your case above, that is immediately followed by
>    --no-whole-archive which counteracts it! I suspect you may have a bug
>    in
>    your pkg-config. By any chance are you using pkg-config 0.27? There is
>    known issue with it not linking static libs correctly, see note in [1]
>    /Bruce
>    [1]
>    [1]https://urldefense.proofpoint.com/v2/url?u=https-3A__doc.dpdk.org_gu
>    ides-2D23.07_linux-5Fgsg_sys-5Freqs.html-23compilation-2Dof-2Dthe-2Ddpd
>    k&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=UaVZAw32qzYwqV0iyjkICkaNhB8peZhhE
>    JSs22LHVbI&m=NHXDGIxEa3NlLGmshhz7un6SMT4NlACKBYA39xVBj3VGki9GCytR-HAHnQ
>    i0cM4S&s=OSlZ0EizFQ8Xa1HkrPrAywHi_kuZHJHhTe48KKtL-AA&e=
> 
> References
> 
>    1. https://urldefense.proofpoint.com/v2/url?u=https-3A__doc.dpdk.org_guides-2D23.07_linux-5Fgsg_sys-5Freqs.html-23compilation-2Dof-2Dthe-2Ddpdk&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=UaVZAw32qzYwqV0iyjkICkaNhB8peZhhEJSs22LHVbI&m=NHXDGIxEa3NlLGmshhz7un6SMT4NlACKBYA39xVBj3VGki9GCytR-HAHnQi0cM4S&s=OSlZ0EizFQ8Xa1HkrPrAywHi_kuZHJHhTe48KKtL-AA&e=