From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 84F49A2EEB for ; Thu, 12 Sep 2019 01:30:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7D0BB1E557; Thu, 12 Sep 2019 01:30:49 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 93DB31D50F; Thu, 12 Sep 2019 01:30:47 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Sep 2019 16:30:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,495,1559545200"; d="scan'208";a="184628800" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga008.fm.intel.com with ESMTP; 11 Sep 2019 16:30:46 -0700 Received: from fmsmsx118.amr.corp.intel.com (10.18.116.18) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 11 Sep 2019 16:30:46 -0700 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by fmsmsx118.amr.corp.intel.com (10.18.116.18) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 11 Sep 2019 16:30:46 -0700 Received: from shsmsx105.ccr.corp.intel.com ([169.254.11.23]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.140]) with mapi id 14.03.0439.000; Thu, 12 Sep 2019 07:30:44 +0800 From: "Zhang, Qi Z" To: "Ye, Xiaolong" CC: "Yigit, Ferruh" , "Loftus, Ciara" , "dev@dpdk.org" , "stable@dpdk.org" , "Karlsson, Magnus" Thread-Topic: [PATCH] net/af_xdp: fix Tx halt when no recv packets Thread-Index: AQHVZyn0j3SRA/CA602neNGCmkpco6ckSvoQgAAe64CAAImBoP//i5KAgAEmvCCAAEZ+gIABL5SQ Date: Wed, 11 Sep 2019 23:30:43 +0000 Message-ID: <039ED4275CED7440929022BC67E7061153D9478C@SHSMSX105.ccr.corp.intel.com> References: <20190909161247.61801-1-xiaolong.ye@intel.com> <039ED4275CED7440929022BC67E7061153D93589@SHSMSX105.ccr.corp.intel.com> <20190910135347.GD6732@intel.com> <039ED4275CED7440929022BC67E7061153D938D9@SHSMSX105.ccr.corp.intel.com> <20190910150913.GE6732@intel.com> <039ED4275CED7440929022BC67E7061153D93F55@SHSMSX105.ccr.corp.intel.com> <20190911125625.GE45267@intel.com> In-Reply-To: <20190911125625.GE45267@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNDMzZTdmYjctNjRjMi00NDQyLWIzYzUtMmI3YmFhZDg4NzA1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiR1ptanFLQUZFYjRiYk5HMTVUT09jNUJhelRTOVZXSjJJRndvZkpJZTA1SjFoYjdhUTMrTEdkVjhSYjR1YlNnOCJ9 x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH] net/af_xdp: fix Tx halt when no recv packets X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Ye, Xiaolong > Sent: Wednesday, September 11, 2019 8:56 PM > To: Zhang, Qi Z > Cc: Yigit, Ferruh ; Loftus, Ciara > ; dev@dpdk.org; stable@dpdk.org; Karlsson, > Magnus > Subject: Re: [PATCH] net/af_xdp: fix Tx halt when no recv packets >=20 > On 09/11, Zhang, Qi Z wrote: > > > > > >> -----Original Message----- > >> From: Ye, Xiaolong > >> Sent: Tuesday, September 10, 2019 11:09 PM > >> To: Zhang, Qi Z > >> Cc: Yigit, Ferruh ; Loftus, Ciara > >> ; dev@dpdk.org; stable@dpdk.org; Karlsson, > >> Magnus > >> Subject: Re: [PATCH] net/af_xdp: fix Tx halt when no recv packets > >> > >> On 09/10, Zhang, Qi Z wrote: > >> > > >> > > >> >> -----Original Message----- > >> >> From: Ye, Xiaolong > >> >> Sent: Tuesday, September 10, 2019 9:54 PM > >> >> To: Zhang, Qi Z > >> >> Cc: Yigit, Ferruh ; Loftus, Ciara > >> >> ; dev@dpdk.org; stable@dpdk.org; Karlsson, > >> >> Magnus > >> >> Subject: Re: [PATCH] net/af_xdp: fix Tx halt when no recv packets > >> >> > >> >> On 09/10, Zhang, Qi Z wrote: > >> >> > > >> >> > > >> >> >> -----Original Message----- > >> >> >> From: Ye, Xiaolong > >> >> >> Sent: Tuesday, September 10, 2019 12:13 AM > >> >> >> To: Yigit, Ferruh ; Loftus, Ciara > >> >> >> ; Ye, Xiaolong ; > >> >> >> Zhang, Qi Z > >> >> >> Cc: dev@dpdk.org; stable@dpdk.org > >> >> >> Subject: [PATCH] net/af_xdp: fix Tx halt when no recv packets > >> >> >> > >> >> >> The kernel only consumes Tx packets if we have some Rx traffic > >> >> >> on specified queue or we have called send(). So we need to > >> >> >> issue a > >> >> >> send() even when the allocation fails so that kernel will start > >> >> >> to consume > >> >> packets again. > >> >> > > >> >> >So "allocation fails" means " xsk_ring_prod__reserve" fail right? > >> >> > >> >> Yes. > >> >> > >> >> >I don't understand when xsk_ring_prod__needs_wakeup is true why > >> >> >kernel will stop Tx packet at this situation would you share more > >> insight? > >> >> > >> >> Actually, the fail case is xsk_ring_prod__needs_wakeup is false, > >> >> then we can't issue a send() when xsk_ring_prod__reserve fails. > >> > > >> >Sorry, I think my question should be for the case when > >> >xsk_ring_prod__needs_wakeup is false, I don't understand why we need > >> >to handle different at below two situations 1. when > >> >xsk_ring_prod__reserve fails 2. normal tx scenario. > >> >My understanding is when xsk_ring_prod__needs_wakeup(tx) is false, > >> which means Tx is ongoing, we don't need to wake up kernel to continue= . > >> > > >> > >> The problem is that kernel does not guarantee that all entries are sen= t for > Tx. > >> There are a number of reasons that this might not happen, but usually > >> some Rx packet will at some point in time in the very short future > >> trigger further Tx processing and the packets will be sent. But if > >> you only have Tx processing and no Rx at all, you have to trigger a se= nto() > again. > > > >Ok , so the question is why we have below code. > >#if defined(XDP_USE_NEED_WAKEUP) > >if (xsk_ring_prod__needs_wakeup(&txq->tx)) > >#endif > > kick_tx(txq); > > > >Here, when xsk_ring_prod__needs_wakeup is false, we can skip kick_tx > (send), but why same "if check" can't be applied to the case when > xsk_ring_prod__reserve failed? >=20 > When the system is running out of Tx entries, it needs some explicit acti= on to > trigger kernel consumes the Tx buffers. >=20 > > > >Btw, think about below case > >when xsk_ring_prod_reserve failed, if we don't kick_tx, and no > >following rx happens, does that mean the remain packets in tx queue will > never get chance be transmitted?, what happen if the last tx_burst is nev= er > be called? >=20 > This is exactly the issue this patch try to fix, in this case, > xsk_ring_prod__reserve failure means there is no more available entries i= n tx > queue, if we don't call send/sendto or there is no rx traffic, Tx just ha= lts. the problem I saw is if a packet is transmitted in a tx_burst, it may get c= hance not be transmitted immediately by kernel (wait for a trigger), it may= rely on a "send" call from next tx_burst.(that's the patch does). But this= does not looks like a correct driver behavior, since we can't assume appli= cation will do tx_burst all the time. I think in every tx_burst we need to guarantee kernel tx should be activate= d immediately, trigger a Tx for a previous tx_burst should not be the case. >=20 > Thanks, > Xiaolong >=20 > > > >> > >> Thanks, > >> Xiaolong > >> > >> >> > >> >> Thanks, > >> >> Xiaolong > >> >> > >> >> > > >> >> >Thanks > >> >> >Qi > >> >> > > >> >> >> > >> >> >> Commit 45bba02c95b0 ("net/af_xdp: support need wakeup > feature") > >> >> >> breaks above rule by adding some condition to send, this patch > >> >> >> fixes it while still keeps the need_wakeup feature for Tx. > >> >> >> > >> >> >> Fixes: 45bba02c95b0 ("net/af_xdp: support need wakeup feature") > >> >> >> Cc: stable@dpdk.org > >> >> >> > >> >> >> Signed-off-by: Xiaolong Ye > >> >> >> --- > >> >> >> drivers/net/af_xdp/rte_eth_af_xdp.c | 28 > >> >> >> ++++++++++++++-------------- > >> >> >> 1 file changed, 14 insertions(+), 14 deletions(-) > >> >> >> > >> >> >> diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c > >> >> >> b/drivers/net/af_xdp/rte_eth_af_xdp.c > >> >> >> index 41ed5b2af..e496e9aaa 100644 > >> >> >> --- a/drivers/net/af_xdp/rte_eth_af_xdp.c > >> >> >> +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c > >> >> >> @@ -286,19 +286,16 @@ kick_tx(struct pkt_tx_queue *txq) { > >> >> >> struct xsk_umem_info *umem =3D txq->pair->umem; > >> >> >> > >> >> >> -#if defined(XDP_USE_NEED_WAKEUP) > >> >> >> - if (xsk_ring_prod__needs_wakeup(&txq->tx)) > >> >> >> -#endif > >> >> >> - while (send(xsk_socket__fd(txq->pair->xsk), NULL, > >> >> >> - 0, MSG_DONTWAIT) < 0) { > >> >> >> - /* some thing unexpected */ > >> >> >> - if (errno !=3D EBUSY && errno !=3D EAGAIN && errno !=3D > >> EINTR) > >> >> >> - break; > >> >> >> - > >> >> >> - /* pull from completion queue to leave more space > */ > >> >> >> - if (errno =3D=3D EAGAIN) > >> >> >> - pull_umem_cq(umem, > >> ETH_AF_XDP_TX_BATCH_SIZE); > >> >> >> - } > >> >> >> + while (send(xsk_socket__fd(txq->pair->xsk), NULL, > >> >> >> + 0, MSG_DONTWAIT) < 0) { > >> >> >> + /* some thing unexpected */ > >> >> >> + if (errno !=3D EBUSY && errno !=3D EAGAIN && errno !=3D > EINTR) > >> >> >> + break; > >> >> >> + > >> >> >> + /* pull from completion queue to leave more space */ > >> >> >> + if (errno =3D=3D EAGAIN) > >> >> >> + pull_umem_cq(umem, > ETH_AF_XDP_TX_BATCH_SIZE); > >> >> >> + } > >> >> >> pull_umem_cq(umem, ETH_AF_XDP_TX_BATCH_SIZE); } > >> >> >> > >> >> >> @@ -367,7 +364,10 @@ eth_af_xdp_tx(void *queue, struct > rte_mbuf > >> >> >> **bufs, uint16_t nb_pkts) > >> >> >> > >> >> >> xsk_ring_prod__submit(&txq->tx, nb_pkts); > >> >> >> > >> >> >> - kick_tx(txq); > >> >> >> +#if defined(XDP_USE_NEED_WAKEUP) > >> >> >> + if (xsk_ring_prod__needs_wakeup(&txq->tx)) > >> >> >> +#endif > >> >> >> + kick_tx(txq); > >> >> >> > >> >> >> txq->stats.tx_pkts +=3D nb_pkts; > >> >> >> txq->stats.tx_bytes +=3D tx_bytes; > >> >> >> -- > >> >> >> 2.17.1 > >> >> >