[dpdk-dev] Secondary process crash in rte_eal_init

[dpdk-dev] Secondary process crash in rte_eal_init

[Dey, Souvik]

Hi All,
                We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.

============================================
Stack Trace of the Crash :
(gdb) bt
#0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
#1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
#2 0x0000000000402545 in __rte_panic ()
#3 0x00000000007353f4 in rte_eal_init ()
#4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
============================================


--
Regards,
Souvik

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Gonzalez Monroy, Sergio]

On 01/04/2015 06:00, Dey, Souvik wrote:
> Hi All,
>                  We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.
>
> ============================================
> Stack Trace of the Crash :
> (gdb) bt
> #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
> #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
> #2 0x0000000000402545 in __rte_panic ()
> #3 0x00000000007353f4 in rte_eal_init ()
> #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
> ============================================
>
>
> --
> Regards,
> Souvik
>
Hi Dey,

Could you try to reproduce the issue with one of the multi-process 
examples from DPDK?
Also you could provide some info such as command line options when 
running both primary/secondary apps, dpdk version, etc...

Sergio

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Burakov, Anatoly]

On 08-Apr-19 2:07 PM, Mohamed Mahmoud (mmahmoud) wrote:
> Hi Anatoly:
> 
> For the secondary process all we are setting is the –proc-type set to 
> secondary, maybe point me to your test that might give me some clues, 
> also How I can generate detailed crash log ?
> 
> When I checked the init code there are very few places that can cause 
> the panic
> 
>    if(pipe(lcore_config[i].pipe_master2slave) < 0)
> 
> rte_panic("Cannot create pipe\n");
> 
> if(pipe(lcore_config[i].pipe_slave2master) < 0)
> 
> rte_panic("Cannot create pipe\n");
> 
>          lcore_config[i].state = WAIT;
> 
> /* create a thread for each lcore */
> 
>          ret = pthread_create(&lcore_config[i].thread_id, NULL,
> 
>                       eal_thread_loop, NULL);
> 
> if(ret != 0)
> 
> rte_panic("Cannot create thread\n");

Hi,

That would be the most recent DPDK version. The issue i was responding 
to was for version 17.08, which has quite a bit more panic's in init code.

In order for me to diagnose the issue i would have to at least know 
where the panic happens. That information is missing from both reports, 
so i cannot begin to imagine what went wrong, let alone why.

> 
> Thanks,
> 
> Mohamed
> 
> *From: *"Burakov, Anatoly" <anatoly.burakov@intel.com>
> *Date: *Monday, April 8, 2019 at 4:42 AM
> *To: *"Jim Holland (jimholla)" <jimholla@cisco.com>, "dev@dpdk.org" 
> <dev@dpdk.org>
> *Cc: *"Mohamed Mahmoud (mmahmoud)" <mmahmoud@cisco.com>, "Min Tang 
> (mtang2)" <mtang2@cisco.com>
> *Subject: *Re: [dpdk-dev] Secondary process crash in rte_eal_init
> 
> On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
> 
>     Hi,
> 
>     We're seeing something similar to what is described in thread below.
>     Our product uses dpdk 17.08. Was there ever a resolution to Souvik's
>     issue?
> 
>     Thanks...Jim
> 
>     On 01/04/2015 06:00, Dey, Souvik wrote:
> 
>         Hi All,
> 
>                             We have a single primary application with
>         multiple secondary applications launched on the same cpu core.
>         When the system boots up for the first time ,the primary comes
>         up followed by the secondary process everything works fine. But
>         in between if I try to restart the secondary processes without
>         restart the primary, the secondary process crashes with the
>         below stack trace. Any idea what is going wrong out here.
> 
>         ============================================
> 
>         Stack Trace of the Crash :
> 
>         (gdb) bt
> 
>         #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
> 
>         #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
> 
>         #2 0x0000000000402545 in __rte_panic ()
> 
>         #3 0x00000000007353f4 in rte_eal_init ()
> 
>         #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
> 
>         ============================================
> 
>         --
> 
>         Regards,
> 
>         Souvik
> 
>     Hi Dey,
> 
>     Could you try to reproduce the issue with one of the multi-process
> 
>     examples from DPDK?
> 
>     Also you could provide some info such as command line options when
> 
>     running both primary/secondary apps, dpdk version, etc...
> 
>     Sergio
> 
> The crash you are seeing is a panic, but i don't see *what* is
> 
> panic'ing, so i cannot diagnose the issue from stack trace alone.
> 
> In general, restarting secondary processes during primary's runtime is a
> 
> supported case, and is indeed the basis for one of our unit tests
> 
> (granted, most secondary process spinups in our unit tests are supposed
> 
> to fail, but some will pass), and i have at various points done
> 
> "secondary process spam" tests firing up tens of thousands of
> 
> secondaries as well. So, crashes on secondary initialization with valid
> 
> parameters is certainly not normal behavior.
> 
> A more detailed crash log would be good to have.
> 
> -- 
> 
> Thanks,
> 
> Anatoly
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Burakov, Anatoly]

On 08-Apr-19 2:07 PM, Mohamed Mahmoud (mmahmoud) wrote:
> Hi Anatoly:
> 
> For the secondary process all we are setting is the –proc-type set to 
> secondary, maybe point me to your test that might give me some clues, 
> also How I can generate detailed crash log ?
> 
> When I checked the init code there are very few places that can cause 
> the panic
> 
>    if(pipe(lcore_config[i].pipe_master2slave) < 0)
> 
> rte_panic("Cannot create pipe\n");
> 
> if(pipe(lcore_config[i].pipe_slave2master) < 0)
> 
> rte_panic("Cannot create pipe\n");
> 
>          lcore_config[i].state = WAIT;
> 
> /* create a thread for each lcore */
> 
>          ret = pthread_create(&lcore_config[i].thread_id, NULL,
> 
>                       eal_thread_loop, NULL);
> 
> if(ret != 0)
> 
> rte_panic("Cannot create thread\n");

Hi,

That would be the most recent DPDK version. The issue i was responding 
to was for version 17.08, which has quite a bit more panic's in init code.

In order for me to diagnose the issue i would have to at least know 
where the panic happens. That information is missing from both reports, 
so i cannot begin to imagine what went wrong, let alone why.

> 
> Thanks,
> 
> Mohamed
> 
> *From: *"Burakov, Anatoly" <anatoly.burakov@intel.com>
> *Date: *Monday, April 8, 2019 at 4:42 AM
> *To: *"Jim Holland (jimholla)" <jimholla@cisco.com>, "dev@dpdk.org" 
> <dev@dpdk.org>
> *Cc: *"Mohamed Mahmoud (mmahmoud)" <mmahmoud@cisco.com>, "Min Tang 
> (mtang2)" <mtang2@cisco.com>
> *Subject: *Re: [dpdk-dev] Secondary process crash in rte_eal_init
> 
> On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
> 
>     Hi,
> 
>     We're seeing something similar to what is described in thread below.
>     Our product uses dpdk 17.08. Was there ever a resolution to Souvik's
>     issue?
> 
>     Thanks...Jim
> 
>     On 01/04/2015 06:00, Dey, Souvik wrote:
> 
>         Hi All,
> 
>                             We have a single primary application with
>         multiple secondary applications launched on the same cpu core.
>         When the system boots up for the first time ,the primary comes
>         up followed by the secondary process everything works fine. But
>         in between if I try to restart the secondary processes without
>         restart the primary, the secondary process crashes with the
>         below stack trace. Any idea what is going wrong out here.
> 
>         ============================================
> 
>         Stack Trace of the Crash :
> 
>         (gdb) bt
> 
>         #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
> 
>         #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
> 
>         #2 0x0000000000402545 in __rte_panic ()
> 
>         #3 0x00000000007353f4 in rte_eal_init ()
> 
>         #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
> 
>         ============================================
> 
>         --
> 
>         Regards,
> 
>         Souvik
> 
>     Hi Dey,
> 
>     Could you try to reproduce the issue with one of the multi-process
> 
>     examples from DPDK?
> 
>     Also you could provide some info such as command line options when
> 
>     running both primary/secondary apps, dpdk version, etc...
> 
>     Sergio
> 
> The crash you are seeing is a panic, but i don't see *what* is
> 
> panic'ing, so i cannot diagnose the issue from stack trace alone.
> 
> In general, restarting secondary processes during primary's runtime is a
> 
> supported case, and is indeed the basis for one of our unit tests
> 
> (granted, most secondary process spinups in our unit tests are supposed
> 
> to fail, but some will pass), and i have at various points done
> 
> "secondary process spam" tests firing up tens of thousands of
> 
> secondaries as well. So, crashes on secondary initialization with valid
> 
> parameters is certainly not normal behavior.
> 
> A more detailed crash log would be good to have.
> 
> -- 
> 
> Thanks,
> 
> Anatoly
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Mohamed Mahmoud (mmahmoud)]

Hi Anatoly:

For the secondary process all we are setting is the –proc-type set to secondary, maybe point me to your test that might give me some clues, also How I can generate detailed crash log ?
When I checked the init code there are very few places that can cause the panic

  if (pipe(lcore_config[i].pipe_master2slave) < 0)
            rte_panic("Cannot create pipe\n");
        if (pipe(lcore_config[i].pipe_slave2master) < 0)
            rte_panic("Cannot create pipe\n");

        lcore_config[i].state = WAIT;

        /* create a thread for each lcore */
        ret = pthread_create(&lcore_config[i].thread_id, NULL,
                     eal_thread_loop, NULL);
        if (ret != 0)
            rte_panic("Cannot create thread\n");


Thanks,
Mohamed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Date: Monday, April 8, 2019 at 4:42 AM
To: "Jim Holland (jimholla)" <jimholla@cisco.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "Mohamed Mahmoud (mmahmoud)" <mmahmoud@cisco.com>, "Min Tang (mtang2)" <mtang2@cisco.com>
Subject: Re: [dpdk-dev] Secondary process crash in rte_eal_init

On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
Hi,
We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?
Thanks...Jim
On 01/04/2015 06:00, Dey, Souvik wrote:
Hi All,
                   We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.

============================================
Stack Trace of the Crash :
(gdb) bt
#0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
#1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
#2 0x0000000000402545 in __rte_panic ()
#3 0x00000000007353f4 in rte_eal_init ()
#4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
============================================


--
Regards,
Souvik

Hi Dey,
Could you try to reproduce the issue with one of the multi-process
examples from DPDK?
Also you could provide some info such as command line options when
running both primary/secondary apps, dpdk version, etc...
Sergio

The crash you are seeing is a panic, but i don't see *what* is
panic'ing, so i cannot diagnose the issue from stack trace alone.

In general, restarting secondary processes during primary's runtime is a
supported case, and is indeed the basis for one of our unit tests
(granted, most secondary process spinups in our unit tests are supposed
to fail, but some will pass), and i have at various points done
"secondary process spam" tests firing up tens of thousands of
secondaries as well. So, crashes on secondary initialization with valid
parameters is certainly not normal behavior.

A more detailed crash log would be good to have.

--
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Mohamed Mahmoud (mmahmoud)]

Hi Anatoly:

For the secondary process all we are setting is the –proc-type set to secondary, maybe point me to your test that might give me some clues, also How I can generate detailed crash log ?
When I checked the init code there are very few places that can cause the panic

  if (pipe(lcore_config[i].pipe_master2slave) < 0)
            rte_panic("Cannot create pipe\n");
        if (pipe(lcore_config[i].pipe_slave2master) < 0)
            rte_panic("Cannot create pipe\n");

        lcore_config[i].state = WAIT;

        /* create a thread for each lcore */
        ret = pthread_create(&lcore_config[i].thread_id, NULL,
                     eal_thread_loop, NULL);
        if (ret != 0)
            rte_panic("Cannot create thread\n");


Thanks,
Mohamed
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
Date: Monday, April 8, 2019 at 4:42 AM
To: "Jim Holland (jimholla)" <jimholla@cisco.com>, "dev@dpdk.org" <dev@dpdk.org>
Cc: "Mohamed Mahmoud (mmahmoud)" <mmahmoud@cisco.com>, "Min Tang (mtang2)" <mtang2@cisco.com>
Subject: Re: [dpdk-dev] Secondary process crash in rte_eal_init

On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
Hi,
We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?
Thanks...Jim
On 01/04/2015 06:00, Dey, Souvik wrote:
Hi All,
                   We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.

============================================
Stack Trace of the Crash :
(gdb) bt
#0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
#1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
#2 0x0000000000402545 in __rte_panic ()
#3 0x00000000007353f4 in rte_eal_init ()
#4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
============================================


--
Regards,
Souvik

Hi Dey,
Could you try to reproduce the issue with one of the multi-process
examples from DPDK?
Also you could provide some info such as command line options when
running both primary/secondary apps, dpdk version, etc...
Sergio

The crash you are seeing is a panic, but i don't see *what* is
panic'ing, so i cannot diagnose the issue from stack trace alone.

In general, restarting secondary processes during primary's runtime is a
supported case, and is indeed the basis for one of our unit tests
(granted, most secondary process spinups in our unit tests are supposed
to fail, but some will pass), and i have at various points done
"secondary process spam" tests firing up tens of thousands of
secondaries as well. So, crashes on secondary initialization with valid
parameters is certainly not normal behavior.

A more detailed crash log would be good to have.

--
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Burakov, Anatoly]

On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
> Hi,
> 
> We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?
> 
> Thanks...Jim
> 
> On 01/04/2015 06:00, Dey, Souvik wrote:
>> Hi All,
>>                   We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.
>>
>> ============================================
>> Stack Trace of the Crash :
>> (gdb) bt
>> #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
>> #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
>> #2 0x0000000000402545 in __rte_panic ()
>> #3 0x00000000007353f4 in rte_eal_init ()
>> #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
>> ============================================
>>
>>
>> --
>> Regards,
>> Souvik
>>
> Hi Dey,
> 
> Could you try to reproduce the issue with one of the multi-process
> examples from DPDK?
> Also you could provide some info such as command line options when
> running both primary/secondary apps, dpdk version, etc...
> 
> Sergio
> 

The crash you are seeing is a panic, but i don't see *what* is 
panic'ing, so i cannot diagnose the issue from stack trace alone.

In general, restarting secondary processes during primary's runtime is a 
supported case, and is indeed the basis for one of our unit tests 
(granted, most secondary process spinups in our unit tests are supposed 
to fail, but some will pass), and i have at various points done 
"secondary process spam" tests firing up tens of thousands of 
secondaries as well. So, crashes on secondary initialization with valid 
parameters is certainly not normal behavior.

A more detailed crash log would be good to have.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Burakov, Anatoly]

On 06-Apr-19 3:26 PM, Jim Holland (jimholla) wrote:
> Hi,
> 
> We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?
> 
> Thanks...Jim
> 
> On 01/04/2015 06:00, Dey, Souvik wrote:
>> Hi All,
>>                   We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.
>>
>> ============================================
>> Stack Trace of the Crash :
>> (gdb) bt
>> #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
>> #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
>> #2 0x0000000000402545 in __rte_panic ()
>> #3 0x00000000007353f4 in rte_eal_init ()
>> #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
>> ============================================
>>
>>
>> --
>> Regards,
>> Souvik
>>
> Hi Dey,
> 
> Could you try to reproduce the issue with one of the multi-process
> examples from DPDK?
> Also you could provide some info such as command line options when
> running both primary/secondary apps, dpdk version, etc...
> 
> Sergio
> 

The crash you are seeing is a panic, but i don't see *what* is 
panic'ing, so i cannot diagnose the issue from stack trace alone.

In general, restarting secondary processes during primary's runtime is a 
supported case, and is indeed the basis for one of our unit tests 
(granted, most secondary process spinups in our unit tests are supposed 
to fail, but some will pass), and i have at various points done 
"secondary process spam" tests firing up tens of thousands of 
secondaries as well. So, crashes on secondary initialization with valid 
parameters is certainly not normal behavior.

A more detailed crash log would be good to have.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Jim Holland (jimholla)]

Hi,

We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?

Thanks...Jim

On 01/04/2015 06:00, Dey, Souvik wrote:
> Hi All,
>                  We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.
>
> ============================================
> Stack Trace of the Crash :
> (gdb) bt
> #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
> #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
> #2 0x0000000000402545 in __rte_panic ()
> #3 0x00000000007353f4 in rte_eal_init ()
> #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
> ============================================
>
>
> --
> Regards,
> Souvik
>
Hi Dey,

Could you try to reproduce the issue with one of the multi-process
examples from DPDK?
Also you could provide some info such as command line options when
running both primary/secondary apps, dpdk version, etc...

Sergio

Re: [dpdk-dev] Secondary process crash in rte_eal_init

[Jim Holland (jimholla)]

Hi,

We're seeing something similar to what is described in thread below. Our product uses dpdk 17.08. Was there ever a resolution to Souvik's issue?

Thanks...Jim

On 01/04/2015 06:00, Dey, Souvik wrote:
> Hi All,
>                  We have a single primary application with multiple secondary applications launched on the same cpu core. When the system boots up for the first time ,the primary comes up followed by the secondary process everything works fine. But in between if I try to restart the secondary processes without restart the primary, the secondary process crashes with the below stack trace. Any idea what is going wrong out here.
>
> ============================================
> Stack Trace of the Crash :
> (gdb) bt
> #0 0x00007f4dad6e01b5 in raise () from /lib/libc.so.6
> #1 0x00007f4dad6e2fc0 in abort () from /lib/libc.so.6
> #2 0x0000000000402545 in __rte_panic ()
> #3 0x00000000007353f4 in rte_eal_init ()
> #4 0x0000000000403133 in main (argc=10, argv=0x7fff2c52fea8)
> ============================================
>
>
> --
> Regards,
> Souvik
>
Hi Dey,

Could you try to reproduce the issue with one of the multi-process
examples from DPDK?
Also you could provide some info such as command line options when
running both primary/secondary apps, dpdk version, etc...

Sergio