Re: [dpdk-dev] ISSUE: compiling with asan+ubsan

[Aaron Conole <aconole@redhat.com>]

"Ananyev, Konstantin" <konstantin.ananyev@intel.com> writes:

>> >> >> While compiling with asan and ubsan I run into the following error:
>> >> >>
>> >> >> FAILED: app/test/app@test@@dpdk-test@exe/test_ring.c.o
>> >> >>
>> >> >> gcc -Iapp/test/app@test@@dpdk-test@exe -Iapp/test -I../app/test
>> >> >> -Ilib/librte_acl -I../lib/librte_acl -I. -I../ -Iconfig -I../config
>> >> >> -
>> >> >> Ilib/librte_eal/include -I../lib/librte_eal/include
>> >> >> -Ilib/librte_eal/linux/include -I../lib/librte_eal/linux/include
>> >> >> -Ilib/librte_eal/common - I../lib/librte_eal/common
>> >> >> -Ilib/librte_eal/x86/include -I../lib/librte_eal/x86/include
>> >> >> -Ilib/librte_eal -I../lib/librte_eal -Ilib/librte_kvargs -
>> >> >> I../lib/librte_kvargs -Ilib/librte_bitratestats
>> >> >> -I../lib/librte_bitratestats -Ilib/librte_ethdev
>> >> >> -I../lib/librte_ethdev -Ilib/librte_net - I../lib/librte_net
>> >> >> -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
>> >> >> -I../lib/librte_mempool -Ilib/librte_ring -I../lib/librte_ring -
>> >> >> Ilib/librte_meter -I../lib/librte_meter -Ilib/librte_metrics
>> >> >> -I../lib/librte_metrics -Ilib/librte_bpf -I../lib/librte_bpf
>> >> >> -Ilib/librte_cfgfile - I../lib/librte_cfgfile -Ilib/librte_cmdline
>> >> >> -I../lib/librte_cmdline -Ilib/librte_cryptodev
>> >> >> -I../lib/librte_cryptodev -Ilib/librte_distributor -
>> >> >> I../lib/librte_distributor -Ilib/librte_efd -I../lib/librte_efd
>> >> >> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_eventdev
>> >> >> -I../lib/librte_eventdev - Ilib/librte_timer -I../lib/librte_timer
>> >> >> -Ilib/librte_fib -I../lib/librte_fib -Ilib/librte_rib
>> >> >> -I../lib/librte_rib -Ilib/librte_flow_classify -
>> >> >> I../lib/librte_flow_classify -Ilib/librte_table -I../lib/librte_table
>> >> >> -Ilib/librte_port -I../lib/librte_port -Ilib/librte_sched
>> >> >> -I../lib/librte_sched - Ilib/librte_ip_frag -I../lib/librte_ip_frag
>> >> >> -Ilib/librte_kni -I../lib/librte_kni -Ilib/librte_pci
>> >> >> -I../lib/librte_pci -Ilib/librte_lpm -I../lib/librte_lpm -
>> >> >> Ilib/librte_ipsec -I../lib/librte_ipsec -Ilib/librte_security
>> >> >> -I../lib/librte_security -Ilib/librte_latencystats
>> >> >> -I../lib/librte_latencystats - Ilib/librte_member
>> >> >> -I../lib/librte_member -Ilib/librte_pipeline -I../lib/librte_pipeline
>> >> >> -Ilib/librte_rawdev -I../lib/librte_rawdev -Ilib/librte_rcu -
>> >> >> I../lib/librte_rcu -Ilib/librte_reorder -I../lib/librte_reorder
>> >> >> -Ilib/librte_stack -I../lib/librte_stack -Idrivers/mempool/ring -
>> >> >> I../drivers/mempool/ring -Idrivers/mempool/stack
>> >> >> -I../drivers/mempool/stack -Idrivers/event/skeleton
>> >> >> -I../drivers/event/skeleton - Idrivers/bus/pci -I../drivers/bus/pci
>> >> >> -I../drivers/bus/pci/linux -Idrivers/bus/vdev -I../drivers/bus/vdev
>> >> >> -Idrivers/net/bonding - I../drivers/net/bonding -Idrivers/net/ring
>> >> >> -I../drivers/net/ring -Ilib/librte_power -I../lib/librte_power
>> >> >> -Ilib/librte_pdump -I../lib/librte_pdump -Ilib/librte_compressdev
>> >> >> -I../lib/librte_compressdev -fdiagnostics-color=always
>> >> >> -fsanitize=address,undefined -fno-omit-frame-pointer -pipe
>> >> >> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2 -g -include
>> >> >> rte_config.h -Wextra -Wcast-qual -Wdeprecated -Wformat- nonliteral
>> >> >> -Wformat-security -Wmissing-declarations -Wmissing-prototypes
>> >> >> -Wnested-externs -Wold-style-definition -Wpointer-arith -
>> >> >> Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings
>> >> >> -Wno-missing-field-initializers -march=native -mno-avx512f -
>> >> >> DALLOW_EXPERIMENTAL_API -Wno-format-truncation -D_GNU_SOURCE -
>> >> MD -MQ
>> >> >> 'app/test/app@test@@dpdk-test@exe/test_ring.c.o'
>> >> >> -MF 'app/test/app@test@@dpdk-test@exe/test_ring.c.o.d' -o
>> >> >> 'app/test/app@test@@dpdk-test@exe/test_ring.c.o' -c
>> >> >> ../app/test/test_ring.c
>> >> >>
>> >> >> cc1: out of memory allocating 65536 bytes after a total of 4609626112
>> >> >> bytes
>> > I have a machine where I hit this error for a lot of other files every
>> > time I do a fresh compile. I attribute that to low swap space.
>> 
>> I am shocked that a translation unit should be 4.6G?  That's
>> unbelievably HUGE - there's no way that's correct.  There must be
>> something going on with your machine - I've only ever seen it with
>> test_ring.c and it's very consistent.
>> 
>> >> >
>> >> > I also noticed that test_ring.c compilation takes a huge amount of time and
>> >> memory.
>> >> > On my box it not as bad as yours (up to ~100 sec and  ~1 GB) , but still
>> > The compilation time issue was attributed to compiler version. We saw
>> > the issues only on clang. GCC did not show any problems.
>> > Is this problem something new?
>> 
>> This is happening on GCC.  Actually, the meson build with clang +
>> asan&ubsan doesn't even work right.
>> 
>> >> seems too much.
>> >> > Will try to have a look later this week.
>> >>
>> >> Okay - glad I'm not the only one.  My only theory is that all the inlining is
>> >> exploding the size of the file during the translation unit processing.
>> >>
>> >> I ran it through the preprocessor and didn't see any large arrays created
>> >> anywhere.  But there are loads of calls to the rte_ring that are tagged as
>> >> "always inline" - even the test_enqueue and test_dequeue functions are
>> >> "always inline" and they are quite large.  Just a completely unfounded guess.
>> 
>> As a test, I changed the attributes in test_ring.h to remove the
>> always_inline - and I was finally able to complete a build with
>> asan&ubsan using GCC.  This is on x86_64 platform.  Just running the
>> fast tests.
>> 
>> Look at test_ring_enqueue/test_ring_dequeue - those functions are large
>> and branchy.  The compiler is being forced to inline them.  I'm not sure
>> how many other places that's happening, but it simply cannot be
>> correct.  They're used all over that test_ring.c translation unit.
>> That's a huge amount of expansion required by the compiler and a bunch
>> of flow analysis being thrown away.  You've pessimized for compilation
>> memory.
>> 
>> Can you please take a closer look?
>
> Can you try:
> http://patches.dpdk.org/patch/69559/
> On my machine it takes less then 10s and 300MB after the patch.

Sure!  Thanks!

>> 
>> >> CC'd Honnappa (and others) to take a look.
>> >>
>> >> >>
>> >> >> This is in a constrained (read: container) environment.  I guess one
>> >> >> way of resolving would be to allocate more memory to the container,
>> >> >> but I'm also curious why the object files are getting so large?
>> >> >> Should I consider this a bug or "working as intended"?  This will
>> >> >> have implications if we want asan/ubsan under the travis build also.
>> >> >>
>> >> >> -Aaron