From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by dpdk.org (Postfix) with ESMTP id 4093029C6 for ; Fri, 16 Nov 2018 11:37:58 +0100 (CET) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Nov 2018 02:37:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,239,1539673200"; d="scan'208";a="86435969" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.221.107]) by fmsmga007.fm.intel.com with SMTP; 16 Nov 2018 02:37:55 -0800 Received: by (sSMTP sendmail emulation); Fri, 16 Nov 2018 10:37:54 +0000 Date: Fri, 16 Nov 2018 10:37:54 +0000 From: Bruce Richardson To: "Lam, Tiago" Cc: dev@dpdk.org, linville@tuxdriver.com Message-ID: <20181116103753.GD3144@bricha3-MOBL.ger.corp.intel.com> References: <24eb3e7e-17f9-222f-aab1-5acfb86823c7@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <24eb3e7e-17f9-222f-aab1-5acfb86823c7@intel.com> Organization: Intel Research and Development Ireland Ltd. User-Agent: Mutt/1.10.1 (2018-07-13) Subject: Re: [dpdk-dev] af_packet dev default "framesz" of 2048B X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2018 10:37:58 -0000 On Thu, Nov 15, 2018 at 07:02:37PM +0000, Lam, Tiago wrote: > Hi guys, > > OvS-DPDK has recently had small a change that changed the data room > available in an mbuf (commit dfaf00e in OvS). This seems to have had the > consequence of breaking the initialisation of eth_af_packets interfaces, > when using default values ("options:dpdk- > devargs=eth_af_packet0,iface=enp61s0f3"). > > After investigating, what seems to be happening is that the > eth_af_packet dev expects an available space of "2048B - TPACKET2_HDRLEN > + sizeof(struct sockaddr_ll) = 2016B" to be available in the data room > of each mbuf. Previous to the above commit, OvS would allocate some > extra space, and this would mean there would be enough room for the > checks performed in eth_rx_queue_setup() and eth_dev_mtu_set() in > rte_eth_af_packet.c. However, with the recent commit that isn't the case > anymore, and without that extra space the first check in > eth_rx_queue_setup() will now be hit and setup of a eth_af_packet > interface fails. > > What I'm trying to understand here is, the logic behind setting a > default 'framesz' of 2048B and it being hardcoded (instead of being > based on the underlying MTU of the interface, or the mbuf data room > directly). The documentation in [1] for mmap() and setting up buffer > rings mentions the exact same values > (tp_block_size=4096,tp_frame_size=2048), which seem to have been > introduced on the first commit, back in 2014. The only constraint > for the framesize, it seems, its that it fits inside the blocksize (i.e. > doesn't span multiple blocksizes), and is aligned to TPACKET_ALIGNMENT. > While I can't comment on the af_packet driver, the reason why in DPDK you need to set your packet buffer size to 2k + headroom, is because the buffers sizes specified to individual NICs can sometimes only be specified in a course-grained manner. For example, if you check https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf for the SRRCTL register, you will see that the buffer size can only be specified in units of 1k. Therefore, when you give the driver a buffer of exactly 2k, and the driver subtracts the headroom space, the actual space writeable by the NIC is below 2k - meaning that the NIC gets told it only has a 1k buffer. This then would lead to 1500-byte packets getting split unnecessarily into two buffers. So the upshot is that any DPDK application needs to allocate buffers of size 2k + HEADROOM + sizeof(struct rte_mbuf). Using buffers of only 2k will not work as expected for some NICs. If OVS is now using 2k buffers, rather than 2k + 256bytes, it will have problems, I think, and should be changed back to using the slightly larger buffer size. Regards, /Bruce