Re: [dpdk-dev] af_packet dev default "framesz" of 2048B

[Bruce Richardson <bruce.richardson@intel.com>]

On Thu, Nov 15, 2018 at 07:02:37PM +0000, Lam, Tiago wrote:
> Hi guys,
> 
> OvS-DPDK has recently had small a change that changed the data room
> available in an mbuf (commit dfaf00e in OvS). This seems to have had the
> consequence of breaking the initialisation of eth_af_packets interfaces,
> when using default values ("options:dpdk-
> devargs=eth_af_packet0,iface=enp61s0f3").
> 
> After investigating, what seems to be happening is that the
> eth_af_packet dev expects an available space of "2048B - TPACKET2_HDRLEN
> + sizeof(struct sockaddr_ll) = 2016B" to be available in the data room
> of each mbuf.  Previous to the above commit, OvS would allocate some
> extra space, and this would mean there would be enough room for the
> checks performed in eth_rx_queue_setup() and eth_dev_mtu_set() in
> rte_eth_af_packet.c. However, with the recent commit that isn't the case
> anymore, and without that extra space the first check in
> eth_rx_queue_setup() will now be hit and setup of a eth_af_packet
> interface fails.
> 
> What I'm trying to understand here is, the logic behind setting a
> default 'framesz' of 2048B and it being hardcoded (instead of being
> based on the underlying MTU of the interface, or the mbuf data room
> directly). The documentation in [1] for mmap() and setting up buffer
> rings mentions the exact same values
> (tp_block_size=4096,tp_frame_size=2048), which seem to have been
> introduced on the first commit, back in 2014. The only constraint
> for the framesize, it seems, its that it fits inside the blocksize (i.e.
> doesn't span multiple blocksizes), and is aligned to TPACKET_ALIGNMENT.
> 

While I can't comment on the af_packet driver, the reason why in DPDK you
need to set your packet buffer size to 2k + headroom, is because the
buffers sizes specified to individual NICs can sometimes only be specified
in a course-grained manner. For example, if you check
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
for the SRRCTL register, you will see that the buffer size can only be
specified in units of 1k. Therefore, when you give the driver a buffer of
exactly 2k, and the driver subtracts the headroom space, the actual space
writeable by the NIC is below 2k - meaning that the NIC gets told it only
has a 1k buffer. This then would lead to 1500-byte packets getting split
unnecessarily into two buffers.

So the upshot is that any DPDK application needs to allocate buffers of
size 2k + HEADROOM + sizeof(struct rte_mbuf). Using buffers of only 2k will
not work as expected for some NICs. If OVS is now using 2k buffers, rather
than 2k + 256bytes, it will have problems, I think, and should be changed
back to using the slightly larger buffer size.

Regards,
/Bruce