[dpdk-dev] Question about DPDK hugepage fd change

[Iain Barker <iain.barker@oracle.com>]

Hi everyone,

We just updated our application from DPDK 17.11.4 (LTS) to DPDK 18.11 (LTS) and we noticed a regression.

Our host platform is providing 2MB huge pages, so for 8GB reservation this means 4000 pages are allocated.

This worked fine in the prior LTS, but after upgrading DPDK what we are seeing is that select() on an fd is failing.

select() works fine when the process starts up, but does not work after DPDK has been initialized.

We did some investigation and found in the DPDK patches linked below, the hugepage tracking mechanism was changed from mmap to an array of file descriptors, and the rlimit for fd's is raised from the default to allow more fd's to be open.

https://mails.dpdk.org/archives/dev/2018-September/110890.html
https://mails.dpdk.org/archives/dev/2018-September/110889.html

The problem is that the GNU C library (glibc) has a limit for the maximum fd passed to select(), and is hard-coded in the POSIX header file and libc at 1024 (and probably many other OS libraries too as a result).

Raising the rlimit for fd >1024 has undefined results, per the manpage:

http://man7.org/linux/man-pages/man2/select.2.html
An fd_set is a fixed size buffer.  Executing FD_CLR() or FD_SET()
with a value of fd that is negative or is equal to or larger than
FD_SETSIZE will result in undefined behavior.  Moreover, POSIX
requires fd to be a valid file descriptor.

The Linux kernel allows file descriptor sets of arbitrary size,
determining the length of the sets to be checked from the value of
nfds.  However, in the glibc implementation, the fd_set type is fixed
in size.

Specifically, libc's header include/sys/select.h has an array of fd's which is FD_SETSIZE deep.
__fd_mask fds_bits[__FD_SETSIZE / __NFDBITS];

and usr/include/linux/posix_types.h is hard-coded with
#define __FD_SETSIZE  1024

As this define and array are in libc, they are used in many libraries on a Linux system. So to use setsize >1024 means recompiling OS libraries and any other package that needs to use FDs, or ensuring that no library used by the application ever calls select() on an fd set. That seems an unreasonable burden...

Any thoughts?

thanks,
Iain