From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com [209.85.220.47]) by dpdk.org (Postfix) with ESMTP id 3E4B0C690 for ; Thu, 30 Jul 2015 19:01:51 +0200 (CEST) Received: by pabkd10 with SMTP id kd10so26250224pab.2 for ; Thu, 30 Jul 2015 10:01:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-type:content-transfer-encoding; bh=akj77Eatvm6qIuR2PXf/t7cp69dJybDyExWHevVi6z8=; b=aIYorx9fqPwpOSwKj+Hu9PYgWiadOJ4ezDIE1gcOtQcqcKW1xVe1UDdYiPlh3HJTK4 tz1QzbPOlhX8a4kbaV0t6ouzYqXxRbxLzRgQ1KjlKCDR2tC2/aXJsmj3Rn6rk4vEhstk Ri/YUNcfI5cm4sajhoynTQ0jFuOpRsVorqm1bUjn5CnmQ2XAriTO975mnoG6snMivNCq XxJfGg6isccoF/leC0fflWY4Hk/FujF4zqCVs07OVI1qtETAdcWdECAFyNDo1CpQb/Fc AHnQjPNjMvoY+Fldm75tdkI/spDXZhj0vr/b/QrSfzZKgiM9LPrikbVwVvf6Uu/3cAXs H+2Q== X-Gm-Message-State: ALoCoQkRw7ziUBVWQKcyxjvB121t8055xVrJ8dmKcxkYYq46EQjshfgMZNQP1zi1EllBnb4baS6h X-Received: by 10.66.230.201 with SMTP id ta9mr108993938pac.95.1438275710591; Thu, 30 Jul 2015 10:01:50 -0700 (PDT) Received: from urahara (static-50-53-82-155.bvtn.or.frontiernet.net. [50.53.82.155]) by smtp.gmail.com with ESMTPSA id hl6sm3157425pdb.28.2015.07.30.10.01.49 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Jul 2015 10:01:50 -0700 (PDT) Date: Thu, 30 Jul 2015 10:01:58 -0700 From: Stephen Hemminger To: Vlad Zolotarov Message-ID: <20150730100158.1516dab3@urahara> In-Reply-To: <55BA55D3.2070105@cloudius-systems.com> References: <55BA3B5D.4020402@cloudius-systems.com> <20150730091753.1af6cc67@urahara> <55BA4EC6.3030301@cloudius-systems.com> <55BA55D3.2070105@cloudius-systems.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] RFC: i40e xmit path HW limitation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 17:01:51 -0000 On Thu, 30 Jul 2015 19:50:27 +0300 Vlad Zolotarov wrote: >=20 >=20 > On 07/30/15 19:20, Avi Kivity wrote: > > > > > > On 07/30/2015 07:17 PM, Stephen Hemminger wrote: > >> On Thu, 30 Jul 2015 17:57:33 +0300 > >> Vlad Zolotarov wrote: > >> > >>> Hi, Konstantin, Helin, > >>> there is a documented limitation of xl710 controllers (i40e driver) > >>> which is not handled in any way by a DPDK driver. > >>> From the datasheet chapter 8.4.1: > >>> > >>> "=E2=80=A2 A single transmit packet may span up to 8 buffers (up to 8= data=20 > >>> descriptors per packet including > >>> both the header and payload buffers). > >>> =E2=80=A2 The total number of data descriptors for the whole TSO (exp= lained=20 > >>> later on in this chapter) is > >>> unlimited as long as each segment within the TSO obeys the previous=20 > >>> rule (up to 8 data descriptors > >>> per segment for both the TSO header and the segment payload buffers)." > >>> > >>> This means that, for instance, long cluster with small fragments has = to > >>> be linearized before it may be placed on the HW ring. > >>> In more standard environments like Linux or FreeBSD drivers the=20 > >>> solution > >>> is straight forward - call skb_linearize()/m_collapse() corresponding. > >>> In the non-conformist environment like DPDK life is not that easy - > >>> there is no easy way to collapse the cluster into a linear buffer from > >>> inside the device driver > >>> since device driver doesn't allocate memory in a fast path and utiliz= es > >>> the user allocated pools only. > >>> > >>> Here are two proposals for a solution: > >>> > >>> 1. We may provide a callback that would return a user TRUE if a give > >>> cluster has to be linearized and it should always be called befo= re > >>> rte_eth_tx_burst(). Alternatively it may be called from inside t= he > >>> rte_eth_tx_burst() and rte_eth_tx_burst() is changed to return=20 > >>> some > >>> error code for a case when one of the clusters it's given has=20 > >>> to be > >>> linearized. > >>> 2. Another option is to allocate a mempool in the driver with the > >>> elements consuming a single page each (standard 2KB buffers would > >>> do). Number of elements in the pool should be as Tx ring length > >>> multiplied by "64KB/(linear data length of the buffer in the pool > >>> above)". Here I use 64KB as a maximum packet length and not taki= ng > >>> into an account esoteric things like "Giant" TSO mentioned in the > >>> spec above. Then we may actually go and linearize the cluster if > >>> needed on top of the buffers from the pool above, post the buffer > >>> from the mempool above on the HW ring, link the original=20 > >>> cluster to > >>> that new cluster (using the private data) and release it when the > >>> send is done. > >> Or just silently drop heavily scattered packets (and increment oerrors) > >> with a PMD_TX_LOG debug message. > >> > >> I think a DPDK driver doesn't have to accept all possible mbufs and do > >> extra work. It seems reasonable to expect caller to be well behaved > >> in this restricted ecosystem. > >> > > > > How can the caller know what's well behaved? It's device dependent. >=20 > +1 >=20 > Stephen, how do you imagine this well-behaved application? Having switch= =20 > case by an underlying device type and then "well-behaving" correspondingl= y? > Not to mention that to "well-behave" the application writer has to read=20 > HW specs and understand them, which would limit the amount of DPDK=20 > developers to a very small amount of people... ;) Not to mention that=20 > the mentioned above switch-case would be a super ugly thing to be found=20 > in an application that would raise a big question about the=20 > justification of a DPDK existence as as SDK providing device drivers=20 > interface. ;) Either have a RTE_MAX_MBUF_SEGMENTS that is global or a mbuf_linearize function? Driver already can stash the mbuf pool used for Rx and reuse it for the transient Tx buffers.