From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5E0EEA04F9; Fri, 10 Jan 2020 11:08:19 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 195381E9C5; Fri, 10 Jan 2020 11:08:19 +0100 (CET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 96D871E9BE for ; Fri, 10 Jan 2020 11:08:17 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Jan 2020 02:08:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,415,1571727600"; d="scan'208";a="212204793" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga007.jf.intel.com with ESMTP; 10 Jan 2020 02:08:16 -0800 Received: from fmsmsx155.amr.corp.intel.com (10.18.116.71) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 10 Jan 2020 02:08:16 -0800 Received: from shsmsx154.ccr.corp.intel.com (10.239.6.54) by FMSMSX155.amr.corp.intel.com (10.18.116.71) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 10 Jan 2020 02:08:15 -0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.30]) by SHSMSX154.ccr.corp.intel.com ([169.254.7.132]) with mapi id 14.03.0439.000; Fri, 10 Jan 2020 18:08:13 +0800 From: "Di, ChenxuX" To: "Ananyev, Konstantin" , "dev@dpdk.org" CC: "Yang, Qiming" Thread-Topic: [dpdk-dev] [PATCH v7 3/4] net/ixgbe: cleanup Tx buffers Thread-Index: AQHVxtj92wSsZt7aBk+5ebKeNY4z4afh1tSAgAHU7hA= Date: Fri, 10 Jan 2020 10:08:13 +0000 Message-ID: <3B926E44943CB04AA3A39AC16328CE39B95155@SHSMSX101.ccr.corp.intel.com> References: <20191203055134.72874-1-chenxux.di@intel.com> <20200109103822.89011-1-chenxux.di@intel.com> <20200109103822.89011-4-chenxux.di@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v7 3/4] net/ixgbe: cleanup Tx buffers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" hi, Konstantin=20 thanks for your opinion, I have fixed almost in new version patch except on= e.=20 > -----Original Message----- > From: Ananyev, Konstantin > Sent: Thursday, January 9, 2020 10:02 PM > To: Di, ChenxuX ; dev@dpdk.org > Cc: Yang, Qiming ; Di, ChenxuX > > Subject: RE: [dpdk-dev] [PATCH v7 3/4] net/ixgbe: cleanup Tx buffers >=20 >=20 > Hi Chenxu, >=20 > Good progress wih _full_version, but still some issues remains I think. > More comments inline. > Konstantin >=20 > > > > Signed-off-by: Chenxu Di > > --- > > drivers/net/ixgbe/ixgbe_ethdev.c | 4 + > > drivers/net/ixgbe/ixgbe_rxtx.c | 156 ++++++++++++++++++++++++++++++- > > drivers/net/ixgbe/ixgbe_rxtx.h | 10 ++ > > 3 files changed, 169 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c > > b/drivers/net/ixgbe/ixgbe_ethdev.c > > index 2c6fd0f13..668c36188 100644 > > --- a/drivers/net/ixgbe/ixgbe_ethdev.c > > +++ b/drivers/net/ixgbe/ixgbe_ethdev.c > > @@ -601,6 +601,7 @@ static const struct eth_dev_ops ixgbe_eth_dev_ops > > =3D { .udp_tunnel_port_add =3D ixgbe_dev_udp_tunnel_port_add, > > .udp_tunnel_port_del =3D ixgbe_dev_udp_tunnel_port_del, > > .tm_ops_get =3D ixgbe_tm_ops_get, > > +.tx_done_cleanup =3D ixgbe_tx_done_cleanup, > > }; > > > > /* > > @@ -649,6 +650,7 @@ static const struct eth_dev_ops ixgbevf_eth_dev_ops > =3D { > > .reta_query =3D ixgbe_dev_rss_reta_query, > > .rss_hash_update =3D ixgbe_dev_rss_hash_update, > > .rss_hash_conf_get =3D ixgbe_dev_rss_hash_conf_get, > > +.tx_done_cleanup =3D ixgbe_tx_done_cleanup, > > }; > > > > /* store statistics names and its offset in stats structure */ @@ > > -1101,6 +1103,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, > > void *init_params __rte_unused) eth_dev->rx_pkt_burst =3D > > &ixgbe_recv_pkts; eth_dev->tx_pkt_burst =3D &ixgbe_xmit_pkts; > > eth_dev->tx_pkt_prepare =3D &ixgbe_prep_pkts; > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_scalar); > > > > /* > > * For secondary processes, we don't initialise any further as > > primary @@ -1580,6 +1583,7 @@ eth_ixgbevf_dev_init(struct rte_eth_dev > > *eth_dev) eth_dev->dev_ops =3D &ixgbevf_eth_dev_ops; > > eth_dev->rx_pkt_burst =3D &ixgbe_recv_pkts; eth_dev->tx_pkt_burst =3D > > &ixgbe_xmit_pkts; > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_scalar); > > > > /* for secondary processes, we don't initialise any further as primary > > * has already done this work. Only check we don't need a different > > diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c > > b/drivers/net/ixgbe/ixgbe_rxtx.c index fa572d184..122dae425 100644 > > --- a/drivers/net/ixgbe/ixgbe_rxtx.c > > +++ b/drivers/net/ixgbe/ixgbe_rxtx.c > > @@ -92,6 +92,8 @@ uint16_t ixgbe_xmit_fixed_burst_vec(void *tx_queue, > struct rte_mbuf **tx_pkts, > > uint16_t nb_pkts); > > #endif > > > > +static ixgbe_tx_done_cleanup_t ixgbe_tx_done_cleanup_op; >=20 > You can't have just one static variable here. > There could be several ixgbe devices and they could be configured in a di= fferent > way. > I.E. txpkt_burst() is per device, so tx_done_cleanup() also has to be per= device. > Probably the easiest way is to add new entry for tx_done_cleanup into str= uct > ixgbe_txq_ops, and set it properly in ixgbe_set_tx_function(). >=20 > > + > > > /**************************************************************** > ***** > > * > > * TX functions > > @@ -2306,6 +2308,152 @@ ixgbe_tx_queue_release_mbufs(struct > > ixgbe_tx_queue *txq) } } > > > > +int > > +ixgbe_tx_done_cleanup_scalar(struct ixgbe_tx_queue *txq, uint32_t > > +free_cnt) >=20 > As a nit I would change _scalar to _full or so. >=20 > > +{ > > +uint32_t pkt_cnt; > > +uint16_t i; > > +uint16_t tx_last; > > +uint16_t tx_id; > > +uint16_t nb_tx_to_clean; > > +uint16_t nb_tx_free_last; > > +struct ixgbe_tx_entry *swr_ring =3D txq->sw_ring; > > + > > +/* Start free mbuf from the next of tx_tail */ tx_last =3D > > +txq->tx_tail; tx_id =3D swr_ring[tx_last].next_id; > > + > > +if (txq->nb_tx_free =3D=3D 0) > > +if (ixgbe_xmit_cleanup(txq)) >=20 >=20 > As a nit it could be just if (ixgbe_set_tx_function && ixgbe_xmit_cleanup= (txq)) >=20 > > +return 0; > > + > > +nb_tx_to_clean =3D txq->nb_tx_free; > > +nb_tx_free_last =3D txq->nb_tx_free; > > +if (!free_cnt) > > +free_cnt =3D txq->nb_tx_desc; > > + > > +/* Loop through swr_ring to count the amount of > > + * freeable mubfs and packets. > > + */ > > +for (pkt_cnt =3D 0; pkt_cnt < free_cnt; ) { for (i =3D 0; i < > > +nb_tx_to_clean && pkt_cnt < free_cnt && tx_id !=3D tx_last; i++) { if > > +(swr_ring[tx_id].mbuf !=3D NULL) { > > +rte_pktmbuf_free_seg(swr_ring[tx_id].mbuf); > > +swr_ring[tx_id].mbuf =3D NULL; > > + > > +/* > > + * last segment in the packet, > > + * increment packet count > > + */ > > +pkt_cnt +=3D (swr_ring[tx_id].last_id =3D=3D tx_id); } > > + > > +tx_id =3D swr_ring[tx_id].next_id; > > +} > > + > > +if (tx_id =3D=3D tx_last || txq->tx_rs_thresh > > +> txq->nb_tx_desc - txq->nb_tx_free) >=20 > First condition (tx_id =3D=3D tx_last) is porbably redundant here. >=20 I think it is necessary. The txq may transmit packets when the API called. So txq->nb_tx_free may be changed. If (tx_id =3D=3D tx_last) , it will break the loop above and the function s= hould be done and return. However if more than txq->tx_rs_thresh numbers packet send into txq while = function doing. It will not return. And fall in endless loop > > +break; > > + > > +if (pkt_cnt < free_cnt) { > > +if (ixgbe_xmit_cleanup(txq)) > > +break; > > + > > +nb_tx_to_clean =3D txq->nb_tx_free - nb_tx_free_last; nb_tx_free_last = =3D > > +txq->nb_tx_free; } } > > + > > +PMD_TX_FREE_LOG(DEBUG, > > +"Free %u Packets successfully " > > +"(port=3D%d queue=3D%d)", > > +pkt_cnt, txq->port_id, txq->queue_id); > > + > > +return (int)pkt_cnt; > > +} > > + > > +int > > +ixgbe_tx_done_cleanup_vec(struct ixgbe_tx_queue *txq __rte_unused, > > +uint32_t free_cnt __rte_unused) { return -ENOTSUP; } > > + > > +int > > +ixgbe_tx_done_cleanup_simple(struct ixgbe_tx_queue *txq, uint32_t > > +free_cnt) { uint16_t i; uint16_t tx_first; uint16_t tx_id; uint32_t > > +pkt_cnt; struct ixgbe_tx_entry *swr_ring =3D txq->sw_ring; >=20 >=20 > Looks overcomplicated here. > TX simple (and vec) doesn't support mulsti-seg packets, So one TXD - one = mbuf, > and one packet. > And ixgbe_tx_free_bufs() always retunrs/frees either 0 or tx_rs_thresh > mbufs/packets. > So it probably can be something like that: >=20 > ixgbe_tx_done_cleanup_simple(struct ixgbe_tx_queue *txq, uint32_t free_c= nt) { > If (free_cnt =3D=3D 0) > free_cnt =3D txq->nb_desc; >=20 > cnt =3D free_cnt - free_cnt % txq->tx_rs_thesh; > for (i =3D 0; i < cnt; i+=3D n) { > n =3D ixgbe_tx_free_bufs(txq); > if (n =3D=3D 0) > break; > } > return i; > } >=20 > > + > > +/* Start free mbuf from tx_first */ > > +tx_first =3D txq->tx_next_dd - (txq->tx_rs_thresh - 1); tx_id =3D > > +tx_first; > > + > > +/* while free_cnt is 0, > > + * suppose one mbuf per packet, > > + * try to free packets as many as possible */ if (free_cnt =3D=3D 0) > > +free_cnt =3D txq->nb_tx_desc; > > + > > +/* Loop through swr_ring to count freeable packets */ for (pkt_cnt =3D > > +0; pkt_cnt < free_cnt; ) { if (txq->nb_tx_desc - txq->nb_tx_free < > > +txq->tx_rs_thresh) break; > > + > > +if (!ixgbe_tx_free_bufs(txq)) > > +break; > > + > > +for (i =3D 0; i !=3D txq->tx_rs_thresh && tx_id !=3D tx_first; i++) { > > +/* last segment in the packet, > > + * increment packet count > > + */ > > +pkt_cnt +=3D (tx_id =3D=3D swr_ring[tx_id].last_id); tx_id =3D > > +swr_ring[tx_id].next_id; } > > + > > +if (tx_id =3D=3D tx_first) > > +break; > > +} > > + > > +PMD_TX_FREE_LOG(DEBUG, > > +"Free %u packets successfully " > > +"(port=3D%d queue=3D%d)", > > +pkt_cnt, txq->port_id, txq->queue_id); > > + > > +return (int)pkt_cnt; > > +} > > + > > +int > > +ixgbe_tx_done_cleanup(void *txq, uint32_t free_cnt) { > > +ixgbe_tx_done_cleanup_t func =3D ixgbe_get_tx_done_cleanup_func(); > > + > > +if (!func) > > +return -ENOTSUP; > > + > > +return func(txq, free_cnt); > > +} > > + > > +void > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_t fn) { > > +ixgbe_tx_done_cleanup_op =3D fn; } > > + > > +ixgbe_tx_done_cleanup_t > > +ixgbe_get_tx_done_cleanup_func(void) > > +{ > > +return ixgbe_tx_done_cleanup_op; > > +} > > + > > static void __attribute__((cold)) > > ixgbe_tx_free_swring(struct ixgbe_tx_queue *txq) { @@ -2398,9 > > +2546,14 @@ ixgbe_set_tx_function(struct rte_eth_dev *dev, struct > > ixgbe_tx_queue *txq) > > ixgbe_txq_vec_setup(txq) =3D=3D 0)) { > > PMD_INIT_LOG(DEBUG, "Vector tx enabled."); dev->tx_pkt_burst =3D > > ixgbe_xmit_pkts_vec; -} else > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_vec); > > +} else { > > #endif > > dev->tx_pkt_burst =3D ixgbe_xmit_pkts_simple; > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_simple); > > +#ifdef RTE_IXGBE_INC_VECTOR > > +} > > +#endif > > } else { > > PMD_INIT_LOG(DEBUG, "Using full-featured tx code path"); > > PMD_INIT_LOG(DEBUG, @@ -2412,6 +2565,7 @@ > ixgbe_set_tx_function(struct > > rte_eth_dev *dev, struct ixgbe_tx_queue *txq) (unsigned > > long)RTE_PMD_IXGBE_TX_MAX_BURST); dev->tx_pkt_burst =3D > > ixgbe_xmit_pkts; dev->tx_pkt_prepare =3D ixgbe_prep_pkts; > > +ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_scalar); > > } > > } > > > > diff --git a/drivers/net/ixgbe/ixgbe_rxtx.h > > b/drivers/net/ixgbe/ixgbe_rxtx.h index 505d344b9..a52597aa9 100644 > > --- a/drivers/net/ixgbe/ixgbe_rxtx.h > > +++ b/drivers/net/ixgbe/ixgbe_rxtx.h > > @@ -253,6 +253,8 @@ struct ixgbe_txq_ops { > > IXGBE_ADVTXD_DCMD_DEXT |\ > > IXGBE_ADVTXD_DCMD_EOP) > > > > +typedef int (*ixgbe_tx_done_cleanup_t)(struct ixgbe_tx_queue *txq, > > +uint32_t free_cnt); > > > > /* Takes an ethdev and a queue and sets up the tx function to be used = based > on > > * the queue parameters. Used in tx_queue_setup by primary process > > and then @@ -285,6 +287,14 @@ int > > ixgbe_rx_vec_dev_conf_condition_check(struct rte_eth_dev *dev); int > > ixgbe_rxq_vec_setup(struct ixgbe_rx_queue *rxq); void > > ixgbe_rx_queue_release_mbufs_vec(struct ixgbe_rx_queue *rxq); > > > > +void ixgbe_set_tx_done_cleanup_func(ixgbe_tx_done_cleanup_t fn); > > +ixgbe_tx_done_cleanup_t ixgbe_get_tx_done_cleanup_func(void); > > + > > +int ixgbe_tx_done_cleanup(void *txq, uint32_t free_cnt); int > > +ixgbe_tx_done_cleanup_scalar(struct ixgbe_tx_queue *txq, uint32_t > > +free_cnt); int ixgbe_tx_done_cleanup_vec(struct ixgbe_tx_queue *txq, > > +uint32_t free_cnt); int ixgbe_tx_done_cleanup_simple(struct > > +ixgbe_tx_queue *txq, uint32_t free_cnt); > > + > > extern const uint32_t ptype_table[IXGBE_PACKET_TYPE_MAX]; > > extern const uint32_t ptype_table_tn[IXGBE_PACKET_TYPE_TN_MAX]; > > > > -- > > 2.17.1 >=20