[dpdk-users] Problems linking DPDK-20.11 libraries to another program

[dpdk-users] Problems linking DPDK-20.11 libraries to another program

[Templin (US), Fred L]

Hello, I am at an impasse trying to link DPDK-20.11 libraries with another program.
The other program (call it "testprog") is a large C source code application that uses
standard linux open source build and compile facilities, including Makefiles, gcc,
configure, etc. It is built from about 190 *.[ch] files, with one defining a single
main() for the entire application.

I read the DPDK documentation and tried to link the libraries in using the
"pkg-config" facility per the recommendations, but I was unable to get pkg-config
to execute out of the "testprog" Makefile. So, I issued the following commands to
obtain the results of running pkg-config:

fltemplin@aer-dtn:~$ pkg-config --cflags libdpdk
fltemplin@aer-dtn:~$ pkg-config --libs libdpdk

I then added the results of the commands to the "testprog" Makefile by adding
the following lines:

CFLAGS += -include rte_config.h -march=native -I/usr/local/include
LDFLAGS += -L/usr/local/lib/x86_64-linux-gnu -Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev -lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs

Next, I modified the DPDK "helloworld" example main.c to link it into "testprog" by
changing the name to "hello.c" and changing the "main()" declaration to a function
call called: "testprog_hello()" which I then called from one of my other "testprog"
C modules. The diffs for how I modified the helloworld code appear at the end of
this message.

I was then able to run "make" in the "testprog" directory, and the build compiled
and appeared to link everything just fine. However, it appears that the resulting
"tesprog" binary only increased in size by about 4% over the size it normally builds
to without the DPDK libraries. This seems at odds with the fact that the DPDK
"dpdk-helloworld" binary is massively large in comparison especially considering
that the helloworld main.c is only 47 lines of C code! The sizes of the binaries
are shown below:

fltemplin@aer-dtn:~$ cd ~/src/testprog/build
fltemplin@aer-dtn:~/src/testprog/build$ ls -l testprog*
-rwxr-xr-x 1 fltemplin fltemplin 4991800 Apr 16 10:25 testprog.nodpdk
-rwxr-xr-x 1 fltemplin fltemplin 5186816 Apr 16 10:32 testprog.withdpdk
fltemplin@aer-dtn~/src/testpog/build$: cd ~/src/DPDK/dpdk-20.11/build/examples/
fltemplin@aer-dtn:~/src/DPDK/dpdk-20.11/build/examples$ ls -l dpdk-helloworld
-rwxr-xr-x 1 fltemplin fltemplin 17834608 Apr 14 14:30 dpdk-helloworld

The next question then came to whether my combined testprog/dpdk code
would run the same as the native DPDK helloworld example. I ultimately want
to set up a NULL network interface when I call "rte_eal_init (argc, argv)" and
so I tried issuing the following invocation of dpdk-helloworld:

fltemplin@aer-dtn:~$ sudo ./dpdk-helloworld --vdev net_null0
EAL: Detected 1 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   Invalid NUMA socket, default to 0
EAL:   Invalid NUMA socket, default to 0
EAL: No legacy callbacks, legacy socket not created
hello from core 0

So, as you can see, dpdk-helloworld passes the "--vnet" and "net_null0" arguments
to "rte_eal_init()", which recognizes the arguments and returns success. But, when
I call "rte_eal_init()" out of my testprog using the same arguments the DPDK service
returns the following:

fltemplin@aer-dtn:~$ sudo ./testprog
EAL: Detected 1 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: failed to parse device "net_null0"
EAL: Unable to parse device 'net_null0'
PANIC in testprog_hello():
Cannot init EAL

This appears to me that DPDK is failing to initialize basic data structures
when run out of my testprog that are initialized fine when run out of the
dpdk-helloworld binary. That the code compiles and links is encouraging,
but the fact that the binary size for my combined testprog/helloworld is
significantly smaller than the size of dpdk-helloworld alone seems very
suspicious. I feel I must be missing something very basic - does anyone
recognize these conditions and have insights about how to fix?

Thanks - Fred

As previously mentioned, see below for my source code changes to link
helloworld into testprog:

*** ~/src/DPDK/dpdk-20.11/examples/helloworld/main.c	2021-04-14 13:59:16.224073175 -0700
--- ~/src/testprog/src/hello.c	2021-04-16 09:46:42.142210166 -0700
***************
*** 2,7 ****
--- 2,14 ----
   * Copyright(c) 2010-2014 Intel Corporation
   */
  
+ #ifdef HAVE_CONFIG_H
+ #include "config.h"
+ #elif defined(_MSC_VER)
+ #include "config-msvc.h"
+ #endif
+ 
+ #ifdef ENABLE_DPDK_TEST
  #include <stdio.h>
  #include <string.h>
  #include <stdint.h>
***************
*** 25,32 ****
--- 32,47 ----
  }
  
  int
+ #ifdef ENABLE_DPDK_TEST
+ testprog_hello()
+ {
+ 	char *args[] = { "openvpn", "--vdev", "net_null0" };
+ 	char **argv = args;
+ 	int argc = 3;
+ #else
  main(int argc, char **argv)
  {
+ #endif
  	int ret;
  	unsigned lcore_id;
  
***************
*** 45,47 ****
--- 60,63 ----
  	rte_eal_mp_wait_lcore();
  	return 0;
  }
+ #endif

Re: [dpdk-users] Problems linking DPDK-20.11 libraries to another program

[Templin (US), Fred L]

All, I just noticed that the "dpdk-helloworld" binary was massively large
because it was built with the "static" option. When I build it as "shared",
the resulting binary is appropriately small. However, when I build it as
"shared" I get the same failure case that I describe for what I am getting
out of my testprog below.

I would greatly prefer to link DPDK into my testprog as "shared" and not
"static". But, if I link it as "shared" is there some sort of pre-initialization
sequence I need to execute before calling "rte_eal_init()"? I would be
happy to be pointed to the appropriate documentation.

Thanks - Fred

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Templin (US), Fred L
> Sent: Friday, April 16, 2021 12:32 PM
> To: users@dpdk.org
> Subject: [dpdk-users] Problems linking DPDK-20.11 libraries to another program
> 
> Hello, I am at an impasse trying to link DPDK-20.11 libraries with another program.
> The other program (call it "testprog") is a large C source code application that uses
> standard linux open source build and compile facilities, including Makefiles, gcc,
> configure, etc. It is built from about 190 *.[ch] files, with one defining a single
> main() for the entire application.
> 
> I read the DPDK documentation and tried to link the libraries in using the
> "pkg-config" facility per the recommendations, but I was unable to get pkg-config
> to execute out of the "testprog" Makefile. So, I issued the following commands to
> obtain the results of running pkg-config:
> 
> fltemplin@aer-dtn:~$ pkg-config --cflags libdpdk
> fltemplin@aer-dtn:~$ pkg-config --libs libdpdk
> 
> I then added the results of the commands to the "testprog" Makefile by adding
> the following lines:
> 
> CFLAGS += -include rte_config.h -march=native -I/usr/local/include
> LDFLAGS += -L/usr/local/lib/x86_64-linux-gnu -Wl,--as-needed -lrte_node -lrte_graph -lrte_bpf -lrte_flow_classify -lrte_pipeline -
> lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_regexdev -
> lrte_rawdev -lrte_pdump -lrte_power -lrte_member -lrte_lpm -lrte_latencystats -lrte_kni -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -
> lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer
> -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu -lrte_ring -lrte_eal
> -lrte_telemetry -lrte_kvargs
> 
> Next, I modified the DPDK "helloworld" example main.c to link it into "testprog" by
> changing the name to "hello.c" and changing the "main()" declaration to a function
> call called: "testprog_hello()" which I then called from one of my other "testprog"
> C modules. The diffs for how I modified the helloworld code appear at the end of
> this message.
> 
> I was then able to run "make" in the "testprog" directory, and the build compiled
> and appeared to link everything just fine. However, it appears that the resulting
> "tesprog" binary only increased in size by about 4% over the size it normally builds
> to without the DPDK libraries. This seems at odds with the fact that the DPDK
> "dpdk-helloworld" binary is massively large in comparison especially considering
> that the helloworld main.c is only 47 lines of C code! The sizes of the binaries
> are shown below:
> 
> fltemplin@aer-dtn:~$ cd ~/src/testprog/build
> fltemplin@aer-dtn:~/src/testprog/build$ ls -l testprog*
> -rwxr-xr-x 1 fltemplin fltemplin 4991800 Apr 16 10:25 testprog.nodpdk
> -rwxr-xr-x 1 fltemplin fltemplin 5186816 Apr 16 10:32 testprog.withdpdk
> fltemplin@aer-dtn~/src/testpog/build$: cd ~/src/DPDK/dpdk-20.11/build/examples/
> fltemplin@aer-dtn:~/src/DPDK/dpdk-20.11/build/examples$ ls -l dpdk-helloworld
> -rwxr-xr-x 1 fltemplin fltemplin 17834608 Apr 14 14:30 dpdk-helloworld
> 
> The next question then came to whether my combined testprog/dpdk code
> would run the same as the native DPDK helloworld example. I ultimately want
> to set up a NULL network interface when I call "rte_eal_init (argc, argv)" and
> so I tried issuing the following invocation of dpdk-helloworld:
> 
> fltemplin@aer-dtn:~$ sudo ./dpdk-helloworld --vdev net_null0
> EAL: Detected 1 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL:   Invalid NUMA socket, default to 0
> EAL:   Invalid NUMA socket, default to 0
> EAL: No legacy callbacks, legacy socket not created
> hello from core 0
> 
> So, as you can see, dpdk-helloworld passes the "--vnet" and "net_null0" arguments
> to "rte_eal_init()", which recognizes the arguments and returns success. But, when
> I call "rte_eal_init()" out of my testprog using the same arguments the DPDK service
> returns the following:
> 
> fltemplin@aer-dtn:~$ sudo ./testprog
> EAL: Detected 1 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: failed to parse device "net_null0"
> EAL: Unable to parse device 'net_null0'
> PANIC in testprog_hello():
> Cannot init EAL
> 
> This appears to me that DPDK is failing to initialize basic data structures
> when run out of my testprog that are initialized fine when run out of the
> dpdk-helloworld binary. That the code compiles and links is encouraging,
> but the fact that the binary size for my combined testprog/helloworld is
> significantly smaller than the size of dpdk-helloworld alone seems very
> suspicious. I feel I must be missing something very basic - does anyone
> recognize these conditions and have insights about how to fix?
> 
> Thanks - Fred
> 
> As previously mentioned, see below for my source code changes to link
> helloworld into testprog:
> 
> *** ~/src/DPDK/dpdk-20.11/examples/helloworld/main.c	2021-04-14 13:59:16.224073175 -0700
> --- ~/src/testprog/src/hello.c	2021-04-16 09:46:42.142210166 -0700
> ***************
> *** 2,7 ****
> --- 2,14 ----
>    * Copyright(c) 2010-2014 Intel Corporation
>    */
> 
> + #ifdef HAVE_CONFIG_H
> + #include "config.h"
> + #elif defined(_MSC_VER)
> + #include "config-msvc.h"
> + #endif
> +
> + #ifdef ENABLE_DPDK_TEST
>   #include <stdio.h>
>   #include <string.h>
>   #include <stdint.h>
> ***************
> *** 25,32 ****
> --- 32,47 ----
>   }
> 
>   int
> + #ifdef ENABLE_DPDK_TEST
> + testprog_hello()
> + {
> + 	char *args[] = { "openvpn", "--vdev", "net_null0" };
> + 	char **argv = args;
> + 	int argc = 3;
> + #else
>   main(int argc, char **argv)
>   {
> + #endif
>   	int ret;
>   	unsigned lcore_id;
> 
> ***************
> *** 45,47 ****
> --- 60,63 ----
>   	rte_eal_mp_wait_lcore();
>   	return 0;
>   }
> + #endif