From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <huawei.xie@intel.com>
Received: from mga14.intel.com (mga14.intel.com [192.55.52.115])
 by dpdk.org (Postfix) with ESMTP id 620D1924E
 for <dev@dpdk.org>; Fri, 23 Oct 2015 04:21:20 +0200 (CEST)
Received: from orsmga001.jf.intel.com ([10.7.209.18])
 by fmsmga103.fm.intel.com with ESMTP; 22 Oct 2015 19:20:55 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.20,185,1444719600"; d="scan'208";a="800745205"
Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201])
 by orsmga001.jf.intel.com with ESMTP; 22 Oct 2015 19:20:55 -0700
Received: from fmsmsx157.amr.corp.intel.com (10.18.116.73) by
 FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS)
 id 14.3.248.2; Thu, 22 Oct 2015 19:20:55 -0700
Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by
 FMSMSX157.amr.corp.intel.com (10.18.116.73) with Microsoft SMTP Server (TLS)
 id 14.3.248.2; Thu, 22 Oct 2015 19:20:54 -0700
Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.96]) by
 SHSMSX152.ccr.corp.intel.com ([169.254.6.106]) with mapi id 14.03.0248.002;
 Fri, 23 Oct 2015 10:20:52 +0800
From: "Xie, Huawei" <huawei.xie@intel.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Thread-Topic: [dpdk-dev] [PATCH v4 6/7] virtio: simple tx routine
Thread-Index: AdENOOrj9g4A3zrLRF2F8uH+Pvbylg==
Date: Fri, 23 Oct 2015 02:20:52 +0000
Message-ID: <C37D651A908B024F974696C65296B57B4B1396B7@SHSMSX101.ccr.corp.intel.com>
References: <1443537953-23917-1-git-send-email-huawei.xie@intel.com>
 <1445515791-25909-1-git-send-email-huawei.xie@intel.com>
 <1445515791-25909-7-git-send-email-huawei.xie@intel.com>
 <20151022095730.5a3c742b@xeon-e3>
 <C37D651A908B024F974696C65296B57B4B139675@SHSMSX101.ccr.corp.intel.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.239.127.40]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH v4 6/7] virtio: simple tx routine
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 23 Oct 2015 02:21:22 -0000

On 10/23/2015 10:17 AM, Xie, Huawei wrote:=0A=
> On 10/23/2015 12:57 AM, Stephen Hemminger wrote:=0A=
>> On Thu, 22 Oct 2015 20:09:50 +0800=0A=
>> Huawei Xie <huawei.xie@intel.com> wrote:=0A=
>>=0A=
>>> Changes in v4:=0A=
>>> - move virtio_xmit_cleanup ahead to free descriptors earlier=0A=
>>>=0A=
>>> Changes in v3:=0A=
>>> - Remove return at the end of void function=0A=
>>> - Remove always_inline attribute for virtio_xmit_cleanup=0A=
>>> bulk free of mbufs when clean used ring.=0A=
>>> shift operation of idx could be saved if vq_free_cnt means=0A=
>>> free slots rather than free descriptors.=0A=
>>>=0A=
>>> TODO: rearrange vq data structure, pack the stats var together so that =
we=0A=
>>> could use one vec instruction to update all of them.=0A=
>>>=0A=
>>> Signed-off-by: Huawei Xie <huawei.xie@intel.com>=0A=
>>> ---=0A=
>>>  drivers/net/virtio/virtio_ethdev.h      |  3 ++=0A=
>>>  drivers/net/virtio/virtio_rxtx_simple.c | 93 +++++++++++++++++++++++++=
++++++++=0A=
>>>  2 files changed, 96 insertions(+)=0A=
>>>=0A=
>>> diff --git a/drivers/net/virtio/virtio_ethdev.h b/drivers/net/virtio/vi=
rtio_ethdev.h=0A=
>>> index d7797ab..ae2d47d 100644=0A=
>>> --- a/drivers/net/virtio/virtio_ethdev.h=0A=
>>> +++ b/drivers/net/virtio/virtio_ethdev.h=0A=
>>> @@ -111,6 +111,9 @@ uint16_t virtio_xmit_pkts(void *tx_queue, struct rt=
e_mbuf **tx_pkts,=0A=
>>>  uint16_t virtio_recv_pkts_vec(void *rx_queue, struct rte_mbuf **rx_pkt=
s,=0A=
>>>  		uint16_t nb_pkts);=0A=
>>>  =0A=
>>> +uint16_t virtio_xmit_pkts_simple(void *tx_queue, struct rte_mbuf **tx_=
pkts,=0A=
>>> +		uint16_t nb_pkts);=0A=
>>> +=0A=
>>>  /*=0A=
>>>   * The VIRTIO_NET_F_GUEST_TSO[46] features permit the host to send us=
=0A=
>>>   * frames larger than 1514 bytes. We do not yet support software LRO=
=0A=
>>> diff --git a/drivers/net/virtio/virtio_rxtx_simple.c b/drivers/net/virt=
io/virtio_rxtx_simple.c=0A=
>>> index ef17562..79b4f7f 100644=0A=
>>> --- a/drivers/net/virtio/virtio_rxtx_simple.c=0A=
>>> +++ b/drivers/net/virtio/virtio_rxtx_simple.c=0A=
>>> @@ -288,6 +288,99 @@ virtio_recv_pkts_vec(void *rx_queue, struct rte_mb=
uf **rx_pkts,=0A=
>>>  	return nb_pkts_received;=0A=
>>>  }=0A=
>>>  =0A=
>>> +#define VIRTIO_TX_FREE_THRESH 32=0A=
>>> +#define VIRTIO_TX_MAX_FREE_BUF_SZ 32=0A=
>>> +#define VIRTIO_TX_FREE_NR 32=0A=
>>> +/* TODO: vq->tx_free_cnt could mean num of free slots so we could avoi=
d shift */=0A=
>>> +static inline void=0A=
>>> +virtio_xmit_cleanup(struct virtqueue *vq)=0A=
>>> +{=0A=
>>> +	uint16_t i, desc_idx;=0A=
>>> +	int nb_free =3D 0;=0A=
>>> +	struct rte_mbuf *m, *free[VIRTIO_TX_MAX_FREE_BUF_SZ];=0A=
>>> +=0A=
>>> +	desc_idx =3D (uint16_t)(vq->vq_used_cons_idx &=0A=
>>> +		((vq->vq_nentries >> 1) - 1));=0A=
>>> +	free[0] =3D (struct rte_mbuf *)vq->vq_descx[desc_idx++].cookie;=0A=
>>> +	nb_free =3D 1;=0A=
>>> +=0A=
>>> +	for (i =3D 1; i < VIRTIO_TX_FREE_NR; i++) {=0A=
>>> +		m =3D (struct rte_mbuf *)vq->vq_descx[desc_idx++].cookie;=0A=
>>> +		if (likely(m->pool =3D=3D free[0]->pool))=0A=
>>> +			free[nb_free++] =3D m;=0A=
>>> +		else {=0A=
>>> +			rte_mempool_put_bulk(free[0]->pool, (void **)free,=0A=
>>> +				nb_free);=0A=
>>> +			free[0] =3D m;=0A=
>>> +			nb_free =3D 1;=0A=
>>> +		}=0A=
>>> +	}=0A=
>>> +=0A=
>>> +	rte_mempool_put_bulk(free[0]->pool, (void **)free, nb_free);=0A=
>>> +	vq->vq_used_cons_idx +=3D VIRTIO_TX_FREE_NR;=0A=
>>> +	vq->vq_free_cnt +=3D (VIRTIO_TX_FREE_NR << 1);=0A=
>>> +}=0A=
>> I think you need to handle refcount, here is a similar patch=0A=
>> for ixgbe.=0A=
> ok, like this:=0A=
>=0A=
> m =3D (struct rte_mbuf *)vq->vq_descx[desc_idx++].cookie;=0A=
Missed a line=0A=
  m =3D __rte_pktmbuf_prefree_seg(m)=0A=
> if (likely(m !=3D NULL)) {=0A=
>     ...=0A=
>=0A=
>> Subject: ixgbe: speed up transmit=0A=
>>=0A=
>> Coalesce transmit buffers and put them back into the pool=0A=
>> in one burst.=0A=
>>=0A=
>> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>=0A=
>>=0A=
>> --- a/lib/librte_pmd_ixgbe/ixgbe_rxtx.c=0A=
>> +++ b/lib/librte_pmd_ixgbe/ixgbe_rxtx.c=0A=
>> @@ -120,12 +120,16 @@ rte_rxmbuf_alloc(struct rte_mempool *mp)=0A=
>>   * Check for descriptors with their DD bit set and free mbufs.=0A=
>>   * Return the total number of buffers freed.=0A=
>>   */=0A=
>> +#define TX_FREE_BULK 32=0A=
>> +=0A=
>>  static inline int __attribute__((always_inline))=0A=
>>  ixgbe_tx_free_bufs(struct ixgbe_tx_queue *txq)=0A=
>>  {=0A=
>>  	struct ixgbe_tx_entry *txep;=0A=
>>  	uint32_t status;=0A=
>> -	int i;=0A=
>> +	int i, n =3D 0;=0A=
>> +	struct rte_mempool *txpool =3D NULL;=0A=
>> +	struct rte_mbuf *free_list[TX_FREE_BULK];=0A=
>>  =0A=
>>  	/* check DD bit on threshold descriptor */=0A=
>>  	status =3D txq->tx_ring[txq->tx_next_dd].wb.status;=0A=
>> @@ -138,20 +142,26 @@ ixgbe_tx_free_bufs(struct ixgbe_tx_queue=0A=
>>  	 */=0A=
>>  	txep =3D &(txq->sw_ring[txq->tx_next_dd - (txq->tx_rs_thresh - 1)]);=
=0A=
>>  =0A=
>> -	/* free buffers one at a time */=0A=
>> -	if ((txq->txq_flags & (uint32_t)ETH_TXQ_FLAGS_NOREFCOUNT) !=3D 0) {=0A=
>> -		for (i =3D 0; i < txq->tx_rs_thresh; ++i, ++txep) {=0A=
>> -			txep->mbuf->next =3D NULL;=0A=
>> -			rte_mempool_put(txep->mbuf->pool, txep->mbuf);=0A=
>> -			txep->mbuf =3D NULL;=0A=
>> -		}=0A=
>> -	} else {=0A=
>> -		for (i =3D 0; i < txq->tx_rs_thresh; ++i, ++txep) {=0A=
>> -			rte_pktmbuf_free_seg(txep->mbuf);=0A=
>> -			txep->mbuf =3D NULL;=0A=
>> +	for (i =3D 0; i < txq->tx_rs_thresh; ++i, ++txep) {=0A=
>> +		struct rte_mbuf *m;=0A=
>> +=0A=
>> +		/* free buffers one at a time */=0A=
>> +		m =3D __rte_pktmbuf_prefree_seg(txep->mbuf);=0A=
>> +		txep->mbuf =3D NULL;=0A=
>> +=0A=
>> +		if (n >=3D TX_FREE_BULK  ||=0A=
> check whether m is NULL here.=0A=
>> +		    (n > 0 && m->pool !=3D txpool)) {=0A=
>> +			rte_mempool_put_bulk(txpool, (void **)free_list, n);=0A=
>> +			n =3D 0;=0A=
>>  		}=0A=
>> +=0A=
>> +		txpool =3D m->pool;=0A=
>> +		free_list[n++] =3D m;=0A=
>>  	}=0A=
>>  =0A=
>> +	if (n > 0)=0A=
>> +		rte_mempool_put_bulk(txpool, (void **)free_list, n);=0A=
>> +=0A=
>>  	/* buffers were freed, update counters */=0A=
>>  	txq->nb_tx_free =3D (uint16_t)(txq->nb_tx_free + txq->tx_rs_thresh);=
=0A=
>>  	txq->tx_next_dd =3D (uint16_t)(txq->tx_next_dd + txq->tx_rs_thresh);=
=0A=
>>=0A=
>>=0A=
>=0A=
=0A=