Re: Including contigmem in core dumps

[Lewis Donzis <lew@perftech.com>]

----- On Oct 22, 2024, at 10:39 AM, Stephen Hemminger stephen@networkplumber.org wrote:

> On Tue, 22 Oct 2024 17:47:11 +0300
> Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> wrote:
> 
>> 2024-10-22 07:41 (UTC-0500), Lewis Donzis:
>> > I've been wondering why we exclude memory allocated by
>> > eal_get_virtual_area() from core dumps? (More specifically, it calls
>> > eal_mem_set_dump() to call madvise() to disable core dumps from the
>> > allocated region.)
>> > 
>> > On many occasions, when debugging after a crash, it would have been very
>> > convenient to be able to see the contents of an mbuf or other object
>> > allocated in contigmem space. And we often avoid using the rte memory
>> > allocator just because of this.
>> > 
>> > Is there any reason for this, or could it perhaps be a compile-time
>> > configuration option not to call madvise()?
>> 
>> The commit that originally added madvise() argued that dumping everything
>> ended up in coredumps with "useless" data [non-mapped or unused pages]:
>> 
>> http://git.dpdk.org/dpdk/commit/?id=d72e4042c5ebda7af81448b387af8218136402d0
>> 
>> Dumping mapped pages sounds reasonable in many cases.
>> Not in all cases admittedly:
>> - legacy memory mode mapping a lot of pages that are not (yet) used;
>> - if packet data is confidential while the app is not.
>> 
>> The option to dump or not can easily be a runtime one.
>> The safe default however seems to be "off".
>> 
>> In dynamic memory node (not FreeSBD, unfortunately)
>> rte_mem_event_callback-register() may be used to call madvise().
>> Maybe DPDK should allow such callbacks in any mode
>> and invoke them during initialization to make the above solution universal.
> 
> It is not unusual to have 2 or 4 Gigabytes of huge pages mapped.
> Many embedded systems do not have 6G of extra storage available for a single
> core
> dump, not to mention multiples. Plus any storage can be really slow on embedded
> systems.
> 
> And the common scenario on Linux is to use systemd to capture and compress
> core dumps.

Totally agree.  Most of the time, we wouldn't want this enabled because the core dumps would be huge, but in environments where we do have storage available, it can greatly help troubleshooting to be able to examine the contents of contigmem in the debugger.  Unfortunately, on FreeBSD, we don't have the luxury of systemd's ability to do this, or even to compress the core dumps, although we do have ZFS compression on the local filesystem, which does a pretty decent job on on core dumps.