Never mind, the problem seems to be with the scheduling policy (and its priority) that I use for the poller. With non-realtime scheduling policies, I can run the poller in busy loop at 100% and not drop a single packet. The bug is invalid.