From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) by dpdk.org (Postfix) with ESMTP id 7ECBDC6BC for ; Thu, 30 Jul 2015 19:56:16 +0200 (CEST) Received: by wibud3 with SMTP id ud3so30464269wib.1 for ; Thu, 30 Jul 2015 10:56:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=iicyrRNdmo+iDFmjGHFqqkdRPdQ/L8tDqhsRTJhLwUs=; b=E/Y9P2ZUVQO0RnueHkvLD4KrsluNxaIO7bTQQ6oium4UhoOP7Occht9YlPq1iwKfwJ +/Mo75RrsS8gnSbWyulhEjQsiHn2KbyHEJ6Z0r+8h8OTZcUQZeFx0+UBaHDnCPjWe40U kLfeia67JBOD193H6A3Rn53sERvm6MBCbIglMER5PZLwqw3KD8YixkUg7YXJw/DuFQyK YJr+vRJ0vukUVpKPfK6SC60ybA3PCCY72C2bs50rFDWmbwztYSeJAC+3sQEFDF7ha0Ap RMHJtzgATqI1SHHwiOTXJKqvQiSBKh9UqwAuDxE9XTnVPHgpP/Q1wimMco6nFEwOTxBc TMng== X-Gm-Message-State: ALoCoQl+HPlHT1FAU313OYeHj++9fokO+e3/1zYcIlAOBD1wNJq8eRtzlup/R8hUtBNo5RFtk9BS X-Received: by 10.194.60.11 with SMTP id d11mr75938898wjr.143.1438278976330; Thu, 30 Jul 2015 10:56:16 -0700 (PDT) Received: from [10.0.0.166] ([37.142.229.250]) by smtp.googlemail.com with ESMTPSA id lz10sm2956598wjb.48.2015.07.30.10.56.15 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Jul 2015 10:56:15 -0700 (PDT) To: "Zhang, Helin" , "Ananyev, Konstantin" References: <55BA3B5D.4020402@cloudius-systems.com> <55BA5468.80109@cloudius-systems.com> From: Vlad Zolotarov Message-ID: <55BA653D.5060109@cloudius-systems.com> Date: Thu, 30 Jul 2015 20:56:13 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] i40e xmit path HW limitation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 17:56:16 -0000 On 07/30/15 20:33, Zhang, Helin wrote: > >> -----Original Message----- >> From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com] >> Sent: Thursday, July 30, 2015 9:44 AM >> To: Zhang, Helin; Ananyev, Konstantin >> Cc: dev@dpdk.org >> Subject: Re: i40e xmit path HW limitation >> >> >> >> On 07/30/15 19:10, Zhang, Helin wrote: >>>> -----Original Message----- >>>> From: Vlad Zolotarov [mailto:vladz@cloudius-systems.com] >>>> Sent: Thursday, July 30, 2015 7:58 AM >>>> To: dev@dpdk.org; Ananyev, Konstantin; Zhang, Helin >>>> Subject: RFC: i40e xmit path HW limitation >>>> >>>> Hi, Konstantin, Helin, >>>> there is a documented limitation of xl710 controllers (i40e driver) >>>> which is not handled in any way by a DPDK driver. >>>> From the datasheet chapter 8.4.1: >>>> >>>> "• A single transmit packet may span up to 8 buffers (up to 8 data >>>> descriptors per packet including both the header and payload buffers). >>>> • The total number of data descriptors for the whole TSO (explained >>>> later on in this chapter) is unlimited as long as each segment within >>>> the TSO obeys the previous rule (up to 8 data descriptors per segment >>>> for both the TSO header and the segment payload buffers)." >>> Yes, I remember the RX side just supports 5 segments per packet receiving. >>> But what's the possible issue you thought about? >> Note that it's a Tx size we are talking about. >> >> See 30520831f058cd9d75c0f6b360bc5c5ae49b5f27 commit in linux net-next repo. >> If such a cluster arrives and you post it on the HW ring - HW will shut this HW ring >> down permanently. The application will see that it's ring is stuck. > That issue was because of using more than 8 descriptors for a packet for TSO. There is no problem in transmitting the TSO packet with more than 8 fragments. On the opposite - one can't transmit a non-TSO packet with more than 8 fragments. One also can't transmit the TSO packet that would contain more than 8 fragments in a single TSO segment including the TSO headers. Pls., read the HW spec as I quoted above for more details. > >>>> This means that, for instance, long cluster with small fragments has to be >>>> linearized before it may be placed on the HW ring. >>> What type of size of the small fragments? Basically 2KB is the default size of >> mbuf of most >>> example applications. 2KB x 8 is bigger than 1.5KB. So it is enough for the >> maximum >>> packet size we supported. >>> If 1KB mbuf is used, don't expect it can transmit more than 8KB size of packet. >> I kinda lost u here. Again, we talk about the Tx side here and buffers >> are not obligatory completely filled. Namely there may be a cluster with >> 15 fragments 100 bytes each. > The root cause is using more than 8 descriptors for a packet. That would be if u would like to SUPER simplify the HW limitation above. In that case u would significantly limit the different packets that may be sent without the linearization. > Linux driver can help > on reducing number of descriptors to be used by merging small size of payload > together, right? > It is not for TSO, it is just for packet transmitting. 2 options in my mind: > 1. Use should ensure it will not use more than 8 descriptors per packet for transmitting. This requirement is too restricting. Pls., see above. > 2. DPDK driver should try to merge small packet together for such case, like Linux kernel driver. > I prefer to use option 1, users should ensure that in the application or up layer software, > and keep the PMD driver as simple as possible. The above statement is super confusing: on the one hand u suggest the DPDK driver to merge the small packet (fragments?) together (how?) and then u immediately propose the user application to do that. Could u, pls., clarify what exactly u suggest here? If that's to leave it to the application - note that it would demand patching all existing DPDK applications that send TCP packets. > > But I have a thought that the maximum number of RX/TX descriptor should be able to be > queried somewhere. There is no such thing as maximum number of Tx fragments in a TSO case. It's only limited by the Tx ring size. > > Regards, > Helin >>>> In more standard environments like Linux or FreeBSD drivers the solution is >>>> straight forward - call skb_linearize()/m_collapse() corresponding. >>>> In the non-conformist environment like DPDK life is not that easy - there is no >>>> easy way to collapse the cluster into a linear buffer from inside the device >> driver >>>> since device driver doesn't allocate memory in a fast path and utilizes the user >>>> allocated pools only. >>>> Here are two proposals for a solution: >>>> >>>> 1. We may provide a callback that would return a user TRUE if a give >>>> cluster has to be linearized and it should always be called before >>>> rte_eth_tx_burst(). Alternatively it may be called from inside the >>>> rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return some >>>> error code for a case when one of the clusters it's given has to be >>>> linearized. >>>> 2. Another option is to allocate a mempool in the driver with the >>>> elements consuming a single page each (standard 2KB buffers would >>>> do). Number of elements in the pool should be as Tx ring length >>>> multiplied by "64KB/(linear data length of the buffer in the pool >>>> above)". Here I use 64KB as a maximum packet length and not taking >>>> into an account esoteric things like "Giant" TSO mentioned in the >>>> spec above. Then we may actually go and linearize the cluster if >>>> needed on top of the buffers from the pool above, post the buffer >>>> from the mempool above on the HW ring, link the original cluster to >>>> that new cluster (using the private data) and release it when the >>>> send is done. >>>> >>>> >>>> The first is a change in the API and would require from the application some >>>> additional handling (linearization). The second would require some additional >>>> memory but would keep all dirty details inside the driver and would leave the >>>> rest of the code intact. >>>> >>>> Pls., comment. >>>> >>>> thanks, >>>> vlad >>>>