From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <vladz@cloudius-systems.com> Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) by dpdk.org (Postfix) with ESMTP id 4ABBFC688 for <dev@dpdk.org>; Thu, 30 Jul 2015 18:50:29 +0200 (CEST) Received: by wibxm9 with SMTP id xm9so76720028wib.1 for <dev@dpdk.org>; Thu, 30 Jul 2015 09:50:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=MLUzcweb6cgiIDRPhmIFt4F3uB0zQc+aPqNISFJ4kxY=; b=Az3n5/irijw+SqkQQmT8jxk+VQMQdsIZJa9o3ROn6tir0UNgyJxJeUjVDXMFmEJUfX wsC+6Vj/+iQnNAuIzq7322MBrudD/eqv6Yg2Cu0HulHUz0PkwdgTUdhD59lffPzzmXRk m9IuX2FXMax3J3DmSehcYKbXKz4kD9mfATv1VrpNjsMJwKdpUlCi500D4I7H/wUWes3J 6L/6iX2dbxmDpschostbZkOvULDJOAsgAFYe5URT7cclWHQTFOv+kvvwtyt/x2s7kfbZ Mi/CkXX6v8d8AS7bb3JIRWwbgZJC2QdPD1vB9QYXfWJa2vvWOu1Vt1nrSgf7GygUVbzW CzKw== X-Gm-Message-State: ALoCoQm3F3bujwd3P/E7q2kG0PRzwbM4OIeEPfxFPXwysblOhguh/Wp7AJuM2cD1IxN+GAkNS2JC X-Received: by 10.180.107.138 with SMTP id hc10mr8250323wib.2.1438275029156; Thu, 30 Jul 2015 09:50:29 -0700 (PDT) Received: from [10.0.0.166] ([37.142.229.250]) by smtp.googlemail.com with ESMTPSA id lg6sm2751897wjb.10.2015.07.30.09.50.28 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Jul 2015 09:50:28 -0700 (PDT) To: Avi Kivity <avi@cloudius-systems.com>, Stephen Hemminger <stephen@networkplumber.org> References: <55BA3B5D.4020402@cloudius-systems.com> <20150730091753.1af6cc67@urahara> <55BA4EC6.3030301@cloudius-systems.com> From: Vlad Zolotarov <vladz@cloudius-systems.com> Message-ID: <55BA55D3.2070105@cloudius-systems.com> Date: Thu, 30 Jul 2015 19:50:27 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55BA4EC6.3030301@cloudius-systems.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] RFC: i40e xmit path HW limitation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 30 Jul 2015 16:50:29 -0000 On 07/30/15 19:20, Avi Kivity wrote: > > > On 07/30/2015 07:17 PM, Stephen Hemminger wrote: >> On Thu, 30 Jul 2015 17:57:33 +0300 >> Vlad Zolotarov <vladz@cloudius-systems.com> wrote: >> >>> Hi, Konstantin, Helin, >>> there is a documented limitation of xl710 controllers (i40e driver) >>> which is not handled in any way by a DPDK driver. >>> From the datasheet chapter 8.4.1: >>> >>> "• A single transmit packet may span up to 8 buffers (up to 8 data >>> descriptors per packet including >>> both the header and payload buffers). >>> • The total number of data descriptors for the whole TSO (explained >>> later on in this chapter) is >>> unlimited as long as each segment within the TSO obeys the previous >>> rule (up to 8 data descriptors >>> per segment for both the TSO header and the segment payload buffers)." >>> >>> This means that, for instance, long cluster with small fragments has to >>> be linearized before it may be placed on the HW ring. >>> In more standard environments like Linux or FreeBSD drivers the >>> solution >>> is straight forward - call skb_linearize()/m_collapse() corresponding. >>> In the non-conformist environment like DPDK life is not that easy - >>> there is no easy way to collapse the cluster into a linear buffer from >>> inside the device driver >>> since device driver doesn't allocate memory in a fast path and utilizes >>> the user allocated pools only. >>> >>> Here are two proposals for a solution: >>> >>> 1. We may provide a callback that would return a user TRUE if a give >>> cluster has to be linearized and it should always be called before >>> rte_eth_tx_burst(). Alternatively it may be called from inside the >>> rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return >>> some >>> error code for a case when one of the clusters it's given has >>> to be >>> linearized. >>> 2. Another option is to allocate a mempool in the driver with the >>> elements consuming a single page each (standard 2KB buffers would >>> do). Number of elements in the pool should be as Tx ring length >>> multiplied by "64KB/(linear data length of the buffer in the pool >>> above)". Here I use 64KB as a maximum packet length and not taking >>> into an account esoteric things like "Giant" TSO mentioned in the >>> spec above. Then we may actually go and linearize the cluster if >>> needed on top of the buffers from the pool above, post the buffer >>> from the mempool above on the HW ring, link the original >>> cluster to >>> that new cluster (using the private data) and release it when the >>> send is done. >> Or just silently drop heavily scattered packets (and increment oerrors) >> with a PMD_TX_LOG debug message. >> >> I think a DPDK driver doesn't have to accept all possible mbufs and do >> extra work. It seems reasonable to expect caller to be well behaved >> in this restricted ecosystem. >> > > How can the caller know what's well behaved? It's device dependent. +1 Stephen, how do you imagine this well-behaved application? Having switch case by an underlying device type and then "well-behaving" correspondingly? Not to mention that to "well-behave" the application writer has to read HW specs and understand them, which would limit the amount of DPDK developers to a very small amount of people... ;) Not to mention that the mentioned above switch-case would be a super ugly thing to be found in an application that would raise a big question about the justification of a DPDK existence as as SDK providing device drivers interface. ;) > >