Re: [dpdk-dev] ISSUE: compiling with asan+ubsan

[Aaron Conole <aconole@redhat.com>]

Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com> writes:

> <snip>
>
>> 
>> "Ananyev, Konstantin" <konstantin.ananyev@intel.com> writes:
>> 
>> > Hi Aaron,
>> >
>> >> While compiling with asan and ubsan I run into the following error:
>> >>
>> >> FAILED: app/test/app@test@@dpdk-test@exe/test_ring.c.o
>> >>
>> >> gcc -Iapp/test/app@test@@dpdk-test@exe -Iapp/test -I../app/test
>> >> -Ilib/librte_acl -I../lib/librte_acl -I. -I../ -Iconfig -I../config
>> >> -
>> >> Ilib/librte_eal/include -I../lib/librte_eal/include
>> >> -Ilib/librte_eal/linux/include -I../lib/librte_eal/linux/include
>> >> -Ilib/librte_eal/common - I../lib/librte_eal/common
>> >> -Ilib/librte_eal/x86/include -I../lib/librte_eal/x86/include
>> >> -Ilib/librte_eal -I../lib/librte_eal -Ilib/librte_kvargs -
>> >> I../lib/librte_kvargs -Ilib/librte_bitratestats
>> >> -I../lib/librte_bitratestats -Ilib/librte_ethdev
>> >> -I../lib/librte_ethdev -Ilib/librte_net - I../lib/librte_net
>> >> -Ilib/librte_mbuf -I../lib/librte_mbuf -Ilib/librte_mempool
>> >> -I../lib/librte_mempool -Ilib/librte_ring -I../lib/librte_ring -
>> >> Ilib/librte_meter -I../lib/librte_meter -Ilib/librte_metrics
>> >> -I../lib/librte_metrics -Ilib/librte_bpf -I../lib/librte_bpf
>> >> -Ilib/librte_cfgfile - I../lib/librte_cfgfile -Ilib/librte_cmdline
>> >> -I../lib/librte_cmdline -Ilib/librte_cryptodev
>> >> -I../lib/librte_cryptodev -Ilib/librte_distributor -
>> >> I../lib/librte_distributor -Ilib/librte_efd -I../lib/librte_efd
>> >> -Ilib/librte_hash -I../lib/librte_hash -Ilib/librte_eventdev
>> >> -I../lib/librte_eventdev - Ilib/librte_timer -I../lib/librte_timer
>> >> -Ilib/librte_fib -I../lib/librte_fib -Ilib/librte_rib
>> >> -I../lib/librte_rib -Ilib/librte_flow_classify -
>> >> I../lib/librte_flow_classify -Ilib/librte_table -I../lib/librte_table
>> >> -Ilib/librte_port -I../lib/librte_port -Ilib/librte_sched
>> >> -I../lib/librte_sched - Ilib/librte_ip_frag -I../lib/librte_ip_frag
>> >> -Ilib/librte_kni -I../lib/librte_kni -Ilib/librte_pci
>> >> -I../lib/librte_pci -Ilib/librte_lpm -I../lib/librte_lpm -
>> >> Ilib/librte_ipsec -I../lib/librte_ipsec -Ilib/librte_security
>> >> -I../lib/librte_security -Ilib/librte_latencystats
>> >> -I../lib/librte_latencystats - Ilib/librte_member
>> >> -I../lib/librte_member -Ilib/librte_pipeline -I../lib/librte_pipeline
>> >> -Ilib/librte_rawdev -I../lib/librte_rawdev -Ilib/librte_rcu -
>> >> I../lib/librte_rcu -Ilib/librte_reorder -I../lib/librte_reorder
>> >> -Ilib/librte_stack -I../lib/librte_stack -Idrivers/mempool/ring -
>> >> I../drivers/mempool/ring -Idrivers/mempool/stack
>> >> -I../drivers/mempool/stack -Idrivers/event/skeleton
>> >> -I../drivers/event/skeleton - Idrivers/bus/pci -I../drivers/bus/pci
>> >> -I../drivers/bus/pci/linux -Idrivers/bus/vdev -I../drivers/bus/vdev
>> >> -Idrivers/net/bonding - I../drivers/net/bonding -Idrivers/net/ring
>> >> -I../drivers/net/ring -Ilib/librte_power -I../lib/librte_power
>> >> -Ilib/librte_pdump -I../lib/librte_pdump -Ilib/librte_compressdev
>> >> -I../lib/librte_compressdev -fdiagnostics-color=always
>> >> -fsanitize=address,undefined -fno-omit-frame-pointer -pipe
>> >> -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Werror -O2 -g -include
>> >> rte_config.h -Wextra -Wcast-qual -Wdeprecated -Wformat- nonliteral
>> >> -Wformat-security -Wmissing-declarations -Wmissing-prototypes
>> >> -Wnested-externs -Wold-style-definition -Wpointer-arith -
>> >> Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings
>> >> -Wno-missing-field-initializers -march=native -mno-avx512f -
>> >> DALLOW_EXPERIMENTAL_API -Wno-format-truncation -D_GNU_SOURCE -
>> MD -MQ
>> >> 'app/test/app@test@@dpdk-test@exe/test_ring.c.o'
>> >> -MF 'app/test/app@test@@dpdk-test@exe/test_ring.c.o.d' -o
>> >> 'app/test/app@test@@dpdk-test@exe/test_ring.c.o' -c
>> >> ../app/test/test_ring.c
>> >>
>> >> cc1: out of memory allocating 65536 bytes after a total of 4609626112
>> >> bytes
> I have a machine where I hit this error for a lot of other files every
> time I do a fresh compile. I attribute that to low swap space.

I am shocked that a translation unit should be 4.6G?  That's
unbelievably HUGE - there's no way that's correct.  There must be
something going on with your machine - I've only ever seen it with
test_ring.c and it's very consistent.

>> >
>> > I also noticed that test_ring.c compilation takes a huge amount of time and
>> memory.
>> > On my box it not as bad as yours (up to ~100 sec and  ~1 GB) , but still
> The compilation time issue was attributed to compiler version. We saw
> the issues only on clang. GCC did not show any problems.
> Is this problem something new?

This is happening on GCC.  Actually, the meson build with clang +
asan&ubsan doesn't even work right.

>> seems too much.
>> > Will try to have a look later this week.
>> 
>> Okay - glad I'm not the only one.  My only theory is that all the inlining is
>> exploding the size of the file during the translation unit processing.
>> 
>> I ran it through the preprocessor and didn't see any large arrays created
>> anywhere.  But there are loads of calls to the rte_ring that are tagged as
>> "always inline" - even the test_enqueue and test_dequeue functions are
>> "always inline" and they are quite large.  Just a completely unfounded guess.

As a test, I changed the attributes in test_ring.h to remove the
always_inline - and I was finally able to complete a build with
asan&ubsan using GCC.  This is on x86_64 platform.  Just running the
fast tests.

Look at test_ring_enqueue/test_ring_dequeue - those functions are large
and branchy.  The compiler is being forced to inline them.  I'm not sure
how many other places that's happening, but it simply cannot be
correct.  They're used all over that test_ring.c translation unit.
That's a huge amount of expansion required by the compiler and a bunch
of flow analysis being thrown away.  You've pessimized for compilation
memory.

Can you please take a closer look?

>> CC'd Honnappa (and others) to take a look.
>> 
>> >>
>> >> This is in a constrained (read: container) environment.  I guess one
>> >> way of resolving would be to allocate more memory to the container,
>> >> but I'm also curious why the object files are getting so large?
>> >> Should I consider this a bug or "working as intended"?  This will
>> >> have implications if we want asan/ubsan under the travis build also.
>> >>
>> >> -Aaron