[dpdk-dev] net/ixgbe: ixgbe_dev_link_update_share() leaks memory and memory mappings due to not cleaning up pthreads

[dpdk-dev] net/ixgbe: ixgbe_dev_link_update_share() leaks memory and memory mappings due to not cleaning up pthreads

[Martin Weiser]

Hi,

we recently discovered that our application was not able to create new
threads or perform mmaps after some time.
With some digging we identified the function
ixgbe_dev_link_update_share() to be the reason for this behavior.

In the patch "net/ixgbe: fix blocking system events"
(819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to run
ixgbe_dev_setup_link_thread_handler().
Unfortunately this thread is not detached and just joined (the very last
one spawned that is) through ixgbe_dev_cancel_link_thread() upon
interface start/stop.

When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
an ixgbe interface with no link this causes a lot of pthreads never to
be cleaned up.
Since each thread holds a mmap to the stack this can quite quickly
exhaust the allowed number of memory mappings for the process.

Best regards,
Martin Weiser

Re: [dpdk-dev] net/ixgbe: ixgbe_dev_link_update_share() leaks memory and memory mappings due to not cleaning up pthreads

[Ananyev, Konstantin]

Hi Martin,

> Sorry, please ignore my previous statement about this having been
> reworked in master. I was comparing to the wrong checkout.
> This issue seems to be still present in the current master.

Can you please file a ticket with your findings in the bugzilla?
Thanks
Konstantin


> 
> Am 09.04.20 um 14:06 schrieb Martin Weiser:
> > Hi,
> >
> > I should have mentioned that our findings apply to DPDK 20.02. I can see
> > in master that this since has been reworked to use rte_eal_alarm_set()
> > instead of using a thread.
> > But maybe this should be addressed in stable?
> >
> > Best regards,
> > Martin Weiser
> >
> >
> > Am 09.04.20 um 12:30 schrieb Martin Weiser:
> >> Hi,
> >>
> >> we recently discovered that our application was not able to create new
> >> threads or perform mmaps after some time.
> >> With some digging we identified the function
> >> ixgbe_dev_link_update_share() to be the reason for this behavior.
> >>
> >> In the patch "net/ixgbe: fix blocking system events"
> >> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to
> >> run ixgbe_dev_setup_link_thread_handler().
> >> Unfortunately this thread is not detached and just joined (the very
> >> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon
> >> interface start/stop.
> >>
> >> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
> >> an ixgbe interface with no link this causes a lot of pthreads never to
> >> be cleaned up.
> >> Since each thread holds a mmap to the stack this can quite quickly
> >> exhaust the allowed number of memory mappings for the process.
> >>
> >> Best regards,
> >> Martin Weiser
> >>

Re: [dpdk-dev] net/ixgbe: ixgbe_dev_link_update_share() leaks memory and memory mappings due to not cleaning up pthreads

[Martin Weiser]

Sorry, please ignore my previous statement about this having been
reworked in master. I was comparing to the wrong checkout.
This issue seems to be still present in the current master.

Am 09.04.20 um 14:06 schrieb Martin Weiser:
> Hi,
>
> I should have mentioned that our findings apply to DPDK 20.02. I can see
> in master that this since has been reworked to use rte_eal_alarm_set()
> instead of using a thread.
> But maybe this should be addressed in stable?
>
> Best regards,
> Martin Weiser
>
>
> Am 09.04.20 um 12:30 schrieb Martin Weiser:
>> Hi,
>>
>> we recently discovered that our application was not able to create new
>> threads or perform mmaps after some time.
>> With some digging we identified the function
>> ixgbe_dev_link_update_share() to be the reason for this behavior.
>>
>> In the patch "net/ixgbe: fix blocking system events"
>> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to
>> run ixgbe_dev_setup_link_thread_handler().
>> Unfortunately this thread is not detached and just joined (the very
>> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon
>> interface start/stop.
>>
>> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
>> an ixgbe interface with no link this causes a lot of pthreads never to
>> be cleaned up.
>> Since each thread holds a mmap to the stack this can quite quickly
>> exhaust the allowed number of memory mappings for the process.
>>
>> Best regards,
>> Martin Weiser
>>

Re: [dpdk-dev] net/ixgbe: ixgbe_dev_link_update_share() leaks memory and memory mappings due to not cleaning up pthreads

[Martin Weiser]

Hi,

I should have mentioned that our findings apply to DPDK 20.02. I can see
in master that this since has been reworked to use rte_eal_alarm_set()
instead of using a thread.
But maybe this should be addressed in stable?

Best regards,
Martin Weiser


Am 09.04.20 um 12:30 schrieb Martin Weiser:
> Hi,
>
> we recently discovered that our application was not able to create new
> threads or perform mmaps after some time.
> With some digging we identified the function
> ixgbe_dev_link_update_share() to be the reason for this behavior.
>
> In the patch "net/ixgbe: fix blocking system events"
> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to
> run ixgbe_dev_setup_link_thread_handler().
> Unfortunately this thread is not detached and just joined (the very
> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon
> interface start/stop.
>
> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
> an ixgbe interface with no link this causes a lot of pthreads never to
> be cleaned up.
> Since each thread holds a mmap to the stack this can quite quickly
> exhaust the allowed number of memory mappings for the process.
>
> Best regards,
> Martin Weiser
>