Hi, we recently discovered that our application was not able to create new threads or perform mmaps after some time. With some digging we identified the function ixgbe_dev_link_update_share() to be the reason for this behavior. In the patch "net/ixgbe: fix blocking system events" (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to run ixgbe_dev_setup_link_thread_handler(). Unfortunately this thread is not detached and just joined (the very last one spawned that is) through ixgbe_dev_cancel_link_thread() upon interface start/stop. When an application repeatedly calls e.g. rte_eth_link_get_nowait() on an ixgbe interface with no link this causes a lot of pthreads never to be cleaned up. Since each thread holds a mmap to the stack this can quite quickly exhaust the allowed number of memory mappings for the process. Best regards, Martin Weiser
Hi,
I should have mentioned that our findings apply to DPDK 20.02. I can see
in master that this since has been reworked to use rte_eal_alarm_set()
instead of using a thread.
But maybe this should be addressed in stable?
Best regards,
Martin Weiser
Am 09.04.20 um 12:30 schrieb Martin Weiser:
> Hi,
>
> we recently discovered that our application was not able to create new
> threads or perform mmaps after some time.
> With some digging we identified the function
> ixgbe_dev_link_update_share() to be the reason for this behavior.
>
> In the patch "net/ixgbe: fix blocking system events"
> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to
> run ixgbe_dev_setup_link_thread_handler().
> Unfortunately this thread is not detached and just joined (the very
> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon
> interface start/stop.
>
> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
> an ixgbe interface with no link this causes a lot of pthreads never to
> be cleaned up.
> Since each thread holds a mmap to the stack this can quite quickly
> exhaust the allowed number of memory mappings for the process.
>
> Best regards,
> Martin Weiser
>
Sorry, please ignore my previous statement about this having been
reworked in master. I was comparing to the wrong checkout.
This issue seems to be still present in the current master.
Am 09.04.20 um 14:06 schrieb Martin Weiser:
> Hi,
>
> I should have mentioned that our findings apply to DPDK 20.02. I can see
> in master that this since has been reworked to use rte_eal_alarm_set()
> instead of using a thread.
> But maybe this should be addressed in stable?
>
> Best regards,
> Martin Weiser
>
>
> Am 09.04.20 um 12:30 schrieb Martin Weiser:
>> Hi,
>>
>> we recently discovered that our application was not able to create new
>> threads or perform mmaps after some time.
>> With some digging we identified the function
>> ixgbe_dev_link_update_share() to be the reason for this behavior.
>>
>> In the patch "net/ixgbe: fix blocking system events"
>> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to
>> run ixgbe_dev_setup_link_thread_handler().
>> Unfortunately this thread is not detached and just joined (the very
>> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon
>> interface start/stop.
>>
>> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on
>> an ixgbe interface with no link this causes a lot of pthreads never to
>> be cleaned up.
>> Since each thread holds a mmap to the stack this can quite quickly
>> exhaust the allowed number of memory mappings for the process.
>>
>> Best regards,
>> Martin Weiser
>>
Hi Martin, > Sorry, please ignore my previous statement about this having been > reworked in master. I was comparing to the wrong checkout. > This issue seems to be still present in the current master. Can you please file a ticket with your findings in the bugzilla? Thanks Konstantin > > Am 09.04.20 um 14:06 schrieb Martin Weiser: > > Hi, > > > > I should have mentioned that our findings apply to DPDK 20.02. I can see > > in master that this since has been reworked to use rte_eal_alarm_set() > > instead of using a thread. > > But maybe this should be addressed in stable? > > > > Best regards, > > Martin Weiser > > > > > > Am 09.04.20 um 12:30 schrieb Martin Weiser: > >> Hi, > >> > >> we recently discovered that our application was not able to create new > >> threads or perform mmaps after some time. > >> With some digging we identified the function > >> ixgbe_dev_link_update_share() to be the reason for this behavior. > >> > >> In the patch "net/ixgbe: fix blocking system events" > >> (819d0d1d57f17aa40321648e99a50b49adc7bdf6) a thread is introduced to > >> run ixgbe_dev_setup_link_thread_handler(). > >> Unfortunately this thread is not detached and just joined (the very > >> last one spawned that is) through ixgbe_dev_cancel_link_thread() upon > >> interface start/stop. > >> > >> When an application repeatedly calls e.g. rte_eth_link_get_nowait() on > >> an ixgbe interface with no link this causes a lot of pthreads never to > >> be cleaned up. > >> Since each thread holds a mmap to the stack this can quite quickly > >> exhaust the allowed number of memory mappings for the process. > >> > >> Best regards, > >> Martin Weiser > >>