Re: [dpdk-dev] [PATCH] ethdev: fix dpdk gcc build on Arm

[Ferruh Yigit <ferruh.yigit@intel.com>]

On 6/8/2020 9:17 AM, Juraj Linkeš wrote:
> 
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> Sent: Thursday, June 4, 2020 3:30 PM
>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; Thomas Monjalon
>> <thomas@monjalon.net>
>> Cc: arybchenko@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli
>> <Honnappa.Nagarahalli@arm.com>
>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm
>>
>> On 6/4/2020 1:55 PM, Juraj Linkeš wrote:
>>>> -----Original Message-----
>>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>> Sent: Thursday, June 4, 2020 2:23 PM
>>>> To: Juraj Linkeš <juraj.linkes@pantheon.tech>; Thomas Monjalon
>>>> <thomas@monjalon.net>
>>>> Cc: arybchenko@solarflare.com; dev@dpdk.org; Honnappa Nagarahalli
>>>> <Honnappa.Nagarahalli@arm.com>
>>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm
>>>>
>>>> On 6/4/2020 11:36 AM, Juraj Linkeš wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>>>> Sent: Wednesday, June 3, 2020 1:41 PM
>>>>>> To: Thomas Monjalon <thomas@monjalon.net>; Juraj Linkeš
>>>>>> <juraj.linkes@pantheon.tech>
>>>>>> Cc: arybchenko@solarflare.com; dev@dpdk.org
>>>>>> Subject: Re: [PATCH] ethdev: fix dpdk gcc build on Arm
>>>>>>
>>>>>> On 6/3/2020 11:16 AM, Thomas Monjalon wrote:
>>>>>>> 03/06/2020 11:48, Juraj Linkeš:
>>>>>>>> Directive #include <file> in gcc implementation searches for
>>>>>>>> files in a standard list of system directories, which leads to a
>>>>>>>> sporadici build error on Taishan arm machines:
>>>>>>>> /tmp/openvpp-testing/dpdk/lib/librte_ethdev/rte_ethdev.h:4287:10:
>>>>>>>> fatal error: rte_ethdev_core.h:
>>>>>>>> No such file or directory #include <rte_ethdev_core.h>
>>>>>>>
>>>>>>> Would be interesting to know why nobody else hit such error?
>>>>>>
>>>>>>
>>>>>> I can't see why this is happening, in the 'mk/rte.lib.mk' we have following:
>>>>>>
>>>>>> "
>>>>>> install: _preinstall build _postinstall
>>>>>> build: _preinstall
>>>>>> "
>>>>>>
>>>>>> Which should cause the library header files installed before
>>>>>> building .c files in that library.
>>>>>> So when compiling 'rte_class_eth.c', the header files should be
>>>>>> already in install folder.
>>>>>>
>>>>>>
>>>>>> I can see how/why changing to "" fixes the issue but I am not sure
>>>>>> about this
>>>> fix.
>>>>>> "rte_ethdev.h" is a public header file, that applications will
>>>>>> include it in their applications. In the public library it is more
>>>>>> proper to have other includes from system folder, using format <>.
>>>>>> Again, I can't see why it is failing but I believe we should find
>>>>>> another solution for _internal_ build error.
>>>>>>
>>>>>>
>>>>>> A very simple solution can be following, but that is also not good,
>>>>>> since it solves the issue by creating a dependency to the order of
>>>>>> the header
>>>> includes:
>>>>>>  diff --git a/lib/librte_ethdev/rte_class_eth.c
>>>>>> b/lib/librte_ethdev/rte_class_eth.c
>>>>>>  index 6338355e25..3030c49020 100644
>>>>>>  --- a/lib/librte_ethdev/rte_class_eth.c
>>>>>>  +++ b/lib/librte_ethdev/rte_class_eth.c
>>>>>>  @@ -10,8 +10,8 @@
>>>>>>   #include <rte_kvargs.h>
>>>>>>   #include <rte_log.h>
>>>>>>
>>>>>>  -#include "rte_ethdev.h"
>>>>>>   #include "rte_ethdev_core.h"
>>>>>>  +#include "rte_ethdev.h"
>>>>>>   #include "rte_ethdev_driver.h"
>>>>>>   #include "ethdev_private.h"
>>>>>>
>>>>>
>>>>> Thomas, Ferruh, what should be the solution? I'm not an expert on
>>>>> this and I
>>>> can't really offer anything better, but I'd like that this gets fixed.
>>>>
>>>> First we need to root cause this before trying to solve it. Honnappa
>>>> also said he can reproduce this but our CI builds can't (we are
>>>> talking about tens of builds daily on various platforms), need to understand
>> why.
>>>> Also from Makefile I can't see how this is happening, I am feeling
>>>> uneasy to fix something before figuring out how/why it is failing.
>>>>
>>>> Can you please try to collect more data on when/how this happens,
>>>> initial questions I can think of:
>>>> - Can you reproduce this with meson build?
>>>> - Is it bare DPDK build, or build part of other project (I guess I
>>>> saw fd.io on the
>>>> link)
>>>> - - If this is not bare DPDK build what changes has been done to build system?
>>>> - Do you see this with fresh build (new clone) or rebuild of existing clone?
>>>> - Can you confirm you have correct RTE_SDK and RTE_TARGET environment
>>>> variables?
>>>> - Can you please share your build command?
>>>>
>>>
>>> I sent an e-mail to dpdk dev a few days back asking for help where I outlined
>> what we're doing:
>>> We're not doing anything special, just downloading and extracting the archive,
>> then setting CONFIG_RTE_LIBRTE_MLX5_PMD and
>> CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC to y in config/common_base and
>> then running make install T=arm64-armv8a-linuxapp-gcc -j. As mentioned in the
>> subject, the build server is a Taishan ARM server.
>>>
>>> We're doing a fresh rebuild everytime. The error doesn't happen everytime,
>> just sometimes - it seems to be random.
>>>
>>> We don't set RTE_SDK nor RTE_TARGET since
>> https://doc.dpdk.org/guides/linux_gsg/build_dpdk.html#installation-of-dpdk-
>> target-environment-using-make doesn't mention those.
>>
>> Can you please try with "V=1",
>> "make install T=arm64-armv8a-linuxapp-gcc -j V=1", this will generate verbose
>> output.
>>
>> And _before_ lines compiling the 'rte_class_eth.c' you should see something like
>> following lines, can you please check:
>> "
>> SYMLINK-FILE include/rte_ethdev_core.h
>>
>> ln -nsf `.../buildtools/relpath.sh .../lib/librte_ethdev/rte_ethdev_core.h
>> .../x86_64-native-linux-gcc/include` .../x86_64-native-linux-gcc/include
>> "
>>
> 
> This is the log with V=1 build (a failed build):
> == Build lib/librte_ethdev
> == Build lib/librte_cmdline
> ar crDs librte_lpm.a rte_lpm.o rte_lpm6.o
>   INSTALL-LIB librte_lpm.a
> ar crDs librte_member.a rte_member.o rte_member_ht.o rte_member_vbf.o
>   INSTALL-LIB librte_member.a
> ar crDs librte_cryptodev.a rte_cryptodev.o rte_cryptodev_pmd.o
>   INSTALL-LIB librte_cryptodev.a
> ar crDs librte_bbdev.a rte_bbdev.o
> ar crDs librte_compressdev.a rte_compressdev.o rte_compressdev_pmd.o rte_comp.o
>   INSTALL-LIB librte_compressdev.a
>   INSTALL-LIB librte_bbdev.a
>   SYMLINK-FILE include/rte_ethdev_pci.h
>   SYMLINK-FILE include/rte_tm.h
>   SYMLINK-FILE include/rte_tm_driver.h
>   SYMLINK-FILE include/rte_eth_ctrl.h
>   SYMLINK-FILE include/rte_flow_driver.h
>   SYMLINK-FILE include/rte_ethdev_vdev.h
>   SYMLINK-FILE include/rte_mtr_driver.h
>   SYMLINK-FILE include/rte_ethdev_core.h
>   SYMLINK-FILE include/rte_flow.h
>   SYMLINK-FILE include/rte_mtr.h
>   SYMLINK-FILE include/rte_ethdev_driver.h
>   SYMLINK-FILE include/rte_ethdev.h
>   SYMLINK-FILE include/rte_dev_info.h
> gcc -Wp,-MD,./.ethdev_private.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o ethdev_private.o -c /tmp/dpdk-20.02/lib/librte_ethdev/ethdev_private.c
> gcc -Wp,-MD,./.rte_ethdev.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o rte_ethdev.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c
> gcc -Wp,-MD,./.rte_class_eth.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o rte_class_eth.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_class_eth.c
> gcc -Wp,-MD,./.rte_flow.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o rte_flow.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_flow.c
> gcc -Wp,-MD,./.rte_tm.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o rte_tm.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_tm.c
>   SYMLINK-FILE include/cmdline.h
>   SYMLINK-FILE include/cmdline_parse.h
>   SYMLINK-FILE include/cmdline_parse_num.h
> gcc -Wp,-MD,./.ethdev_profile.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o ethdev_profile.o -c /tmp/dpdk-20.02/lib/librte_ethdev/ethdev_profile.c
> gcc -Wp,-MD,./.rte_mtr.o.d.tmp   -pthread -I/tmp/dpdk-20.02/lib/librte_eal/linux/eal/include  -march=armv8-a+crc -DRTE_MACHINE_CPUFLAG_NEON -DRTE_MACHINE_CPUFLAG_CRC32  -I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include -DRTE_USE_FUNCTION_VERSIONING -include /tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include/rte_config.h -D_GNU_SOURCE -DALLOW_EXPERIMENTAL_API -O3 -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wold-style-definition -Wpointer-arith -Wcast-align -Wnested-externs -Wcast-qual -Wformat-nonliteral -Wformat-security -Wundef -Wwrite-strings -Wdeprecated -Wno-missing-field-initializers -Wimplicit-fallthrough=2 -Wno-format-truncation -Wno-address-of-packed-member    -o rte_mtr.o -c /tmp/dpdk-20.02/lib/librte_ethdev/rte_mtr.c
>   SYMLINK-FILE include/rte_sched.h
>   SYMLINK-FILE include/rte_red.h
>   SYMLINK-FILE include/rte_approx.h
> In file included from /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev.c:43:
> /tmp/dpdk-20.02/lib/librte_ethdev/rte_ethdev_driver.h:18:10: fatal error: rte_ethdev.h: No such file or directory
>  #include <rte_ethdev.h>
>           ^~~~~~~~~~~~~~
> compilation terminated.
> /tmp/dpdk-20.02/mk/internal/rte.compile-pre.mk:114: recipe for target 'rte_ethdev.o' failed
> 
>> Also for the failed case, after fail can you please check if
>> a) '.../arm64-armv8a-linuxapp-gcc/build/lib/librte_ethdev/_preinstall' file exists
>> b) '.../x86_64-native-linux-gcc/include/rte_ethdev_core.h' exists and it linked to
>> correct file '../../lib/librte_ethdev/rte_ethdev_core.h'
>>
> 
> a) it doesn't exist
> b) all of these exist:
> rte_ethdev_core.h -> ../../lib/librte_ethdev/rte_ethdev_core.h
> rte_ethdev_driver.h -> ../../lib/librte_ethdev/rte_ethdev_driver.h
> rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h
> rte_ethdev_pci.h -> ../../lib/librte_ethdev/rte_ethdev_pci.h
> rte_ethdev_vdev.h -> ../../lib/librte_ethdev/rte_ethdev_vdev.h
> 
> I was able to reproduce the failure after 7 successful builds.

Thanks for details.

This time it seems build complaining about not able to find "rte_ethdev.h", so
your patch won't fix this case and build still will fail, looks like we need
different fix.

And in the build command for 'rte_ethdev.c', as you can see there is
'-I/tmp/dpdk-20.02/arm64-armv8a-linuxapp-gcc/include' paramter, and you
mentioned above "rte_ethdev.h -> ../../lib/librte_ethdev/rte_ethdev.h" link
exist under that filder, so still not clear why compiler can't find the files.

There is a dependency of build for the '_preinstall' file, so buildling .c files
in that library shouldn't start before all header files are installed and
'_preinstall' file is created. This is also not clear how '_preinstall' can be
missing.

Can you please double check above links are valid, I mean you can access to the
files from links?

> 
>>
>>>
>>> I'll try Meson build a few times. How can I enable those two config options in
>> Meson?
>>>
>>>> Thanks,
>>>> ferruh
>>>>
>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Fix this by replacing the directive with #include "file" which
>>>>>>>> searches for files first in the directory containing the current
>>>>>>>> file and other directories only after that.
>>>>>>>>
>>>>>>>> Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
>>>>>>>
>>>>>>> What about #include <rte_ethdev_trace.h> ?
>>>>>>>
>>>>>>> Except this,
>>>>>>> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>>>>>>>
>>>>>>>
>>>>>>> I would change the title to be more generic (nothing specific to
>>>>>>> gcc or Arm
>>>>>> here):
>>>>>>> 	ethdev: fix local include
>>>>>>>
>>>>>>> It needs to be backported, so these lines are required:
>>>>>>> 	Fixes: 7f0a669e7b04 ("ethdev: add allocation helper for virtual drivers")
>>>>>>> 	Fixes: dcd5c8112bc3 ("ethdev: add PCI driver helpers")
>>>>>>> 	Fixes: ffc905f3b856 ("ethdev: separate driver APIs")
>>>>>>> 	Fixes: 331c447ad913 ("ethdev: separate internal structures into
>>>>>>> own
>>>>>> header")
>>>>>>> 	Cc: stable@dpdk.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>