Questions on i40e TX path

Questions on i40e TX path

[Honnappa Nagarahalli]

Hi Beilei,
	I want to make sure my understanding of the TX path is correct. Following is my understanding.

1) The RS bit must be set in the TX descriptors to ask the NIC to report back the send status.
2) The NIC reports the send completion by setting the DTYPE field to 0xf. This also indicates that all the earlier descriptors are also done sending the packets.
3) The check "if (txq->nb_tx_free < txq->tx_free_thresh)" is mainly to ensure that we do not check the "descriptor done" status too often.

Is my understanding correct?

Thank you,
Honnappa

RE: Questions on i40e TX path

[Xing, Beilei]

Hi Honnappa,

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Wednesday, February 9, 2022 12:36 PM
> To: Xing, Beilei <beilei.xing@intel.com>; dev@dpdk.org
> Cc: Feifei Wang <Feifei.Wang2@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Richardson,
> Bruce <bruce.richardson@intel.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: Questions on i40e TX path
> 
> Hi Beilei,
> 	I want to make sure my understanding of the TX path is correct.
> Following is my understanding.
> 
> 1) The RS bit must be set in the TX descriptors to ask the NIC to report back the
> send status.
Not for each Tx descriptor.
According to the datasheet, " The RS flag can be set only on a last Transmit Data
Descriptor of a packet or last Transmit Data Descriptor of a TSO or last Transmit
Data Descriptor of a filter." 

> 2) The NIC reports the send completion by setting the DTYPE field to 0xf. This
> also indicates that all the earlier descriptors are also done sending the packets.
Yes.

> 3) The check "if (txq->nb_tx_free < txq->tx_free_thresh)" is mainly to ensure that
> we do not check the "descriptor done" status too often.
This condition is to ensure there're enough free descriptors for Tx, avoid Tx ring full.

> 
> Is my understanding correct?
> 
> Thank you,
> Honnappa

RE: Questions on i40e TX path

[Honnappa Nagarahalli]

<snip>

Thank you for your input. Please see few comments inline.

> > Subject: Questions on i40e TX path
> >
> > Hi Beilei,
> > 	I want to make sure my understanding of the TX path is correct.
> > Following is my understanding.
> >
> > 1) The RS bit must be set in the TX descriptors to ask the NIC to
> > report back the send status.
> Not for each Tx descriptor.
> According to the datasheet, " The RS flag can be set only on a last Transmit Data
> Descriptor of a packet or last Transmit Data Descriptor of a TSO or last Transmit
> Data Descriptor of a filter."
Yes, understood.
When combined with #2 below, we are asking the NIC to report back the send/completion status for a set of packets. This allows for amortization of cost of reporting over the set of packets.

> 
> > 2) The NIC reports the send completion by setting the DTYPE field to
> > 0xf. This also indicates that all the earlier descriptors are also done sending
> the packets.
> Yes.
> 
> > 3) The check "if (txq->nb_tx_free < txq->tx_free_thresh)" is mainly to
> > ensure that we do not check the "descriptor done" status too often.
> This condition is to ensure there're enough free descriptors for Tx, avoid Tx ring
> full.
Ok. I think this check has another purpose as well, though I am not sure if it is intentional. I see that the descriptors are initialized with DTYPE set to 0xF (in function i40e_reset_tx_queue). So, in the very first call to transmit function (for ex: i40e_xmit_fixed_burst_vec), the 'i40e_tx_free_bufs' function would end up checking the DTYPE field, if the above check was not there.

In the data sheet, in section 8.4.2.1.1 (transmit data descriptor format), the RS field is described as follows:

" Report Status. When set, the hardware reports the DMA completion of the transmit descriptor and its
data buffer. Completion is reported by descriptor write back or by head write back as configured by the
HEAD_WBEN flag in the transmit context. When it is reported by descriptor write back, the DTYP field is
set to 0xF and the RS flag is set."

Considering the last sentence, should the code check for both DTYP field and RS field for completion? Currently, the code checks for just the DTYP field (in the function i40e_tx_free_bufs).

> 
> >
> > Is my understanding correct?
> >
> > Thank you,
> > Honnappa

RE: Questions on i40e TX path

[Xing, Beilei]

> -----Original Message-----
> From: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>
> Sent: Thursday, February 10, 2022 12:38 PM
> To: Xing, Beilei <beilei.xing@intel.com>; dev@dpdk.org
> Cc: Feifei Wang <Feifei.Wang2@arm.com>; Ruifeng Wang
> <Ruifeng.Wang@arm.com>; Yigit, Ferruh <ferruh.yigit@intel.com>; Richardson,
> Bruce <bruce.richardson@intel.com>; nd <nd@arm.com>; nd <nd@arm.com>
> Subject: RE: Questions on i40e TX path
> 
> <snip>
> 
> Thank you for your input. Please see few comments inline.
> 
> > > Subject: Questions on i40e TX path
> > >
> > > Hi Beilei,
> > > 	I want to make sure my understanding of the TX path is correct.
> > > Following is my understanding.
> > >
> > > 1) The RS bit must be set in the TX descriptors to ask the NIC to
> > > report back the send status.
> > Not for each Tx descriptor.
> > According to the datasheet, " The RS flag can be set only on a last
> > Transmit Data Descriptor of a packet or last Transmit Data Descriptor
> > of a TSO or last Transmit Data Descriptor of a filter."
> Yes, understood.
> When combined with #2 below, we are asking the NIC to report back the
> send/completion status for a set of packets. This allows for amortization of cost
> of reporting over the set of packets.
> 
> >
> > > 2) The NIC reports the send completion by setting the DTYPE field to
> > > 0xf. This also indicates that all the earlier descriptors are also
> > > done sending
> > the packets.
> > Yes.
> >
> > > 3) The check "if (txq->nb_tx_free < txq->tx_free_thresh)" is mainly
> > > to ensure that we do not check the "descriptor done" status too often.
> > This condition is to ensure there're enough free descriptors for Tx,
> > avoid Tx ring full.
> Ok. I think this check has another purpose as well, though I am not sure if it is
> intentional. I see that the descriptors are initialized with DTYPE set to 0xF (in
> function i40e_reset_tx_queue). So, in the very first call to transmit function (for
> ex: i40e_xmit_fixed_burst_vec), the 'i40e_tx_free_bufs' function would end up
> checking the DTYPE field, if the above check was not there.
> 
> In the data sheet, in section 8.4.2.1.1 (transmit data descriptor format), the RS
> field is described as follows:
> 
> " Report Status. When set, the hardware reports the DMA completion of the
> transmit descriptor and its data buffer. Completion is reported by descriptor
> write back or by head write back as configured by the HEAD_WBEN flag in the
> transmit context. When it is reported by descriptor write back, the DTYP field is
> set to 0xF and the RS flag is set."
> 
> Considering the last sentence, should the code check for both DTYP field and RS
> field for completion? Currently, the code checks for just the DTYP field (in the
> function i40e_tx_free_bufs).

In my understanding, RS bit is set by SW to indicate HW needs to write back. So I don't
think PMD needs to check RS bit.
Besides, PMD should have ensured PMD checks DTYPE of the descriptor with RS bit only.

> 
> >
> > >
> > > Is my understanding correct?
> > >
> > > Thank you,
> > > Honnappa

RE: Questions on i40e TX path

[Honnappa Nagarahalli]

<snip>

> >
> > Thank you for your input. Please see few comments inline.
> >
> > > > Subject: Questions on i40e TX path
> > > >
> > > > Hi Beilei,
> > > > 	I want to make sure my understanding of the TX path is correct.
> > > > Following is my understanding.
> > > >
> > > > 1) The RS bit must be set in the TX descriptors to ask the NIC to
> > > > report back the send status.
> > > Not for each Tx descriptor.
> > > According to the datasheet, " The RS flag can be set only on a last
> > > Transmit Data Descriptor of a packet or last Transmit Data
> > > Descriptor of a TSO or last Transmit Data Descriptor of a filter."
> > Yes, understood.
> > When combined with #2 below, we are asking the NIC to report back the
> > send/completion status for a set of packets. This allows for
> > amortization of cost of reporting over the set of packets.
> >
> > >
> > > > 2) The NIC reports the send completion by setting the DTYPE field
> > > > to 0xf. This also indicates that all the earlier descriptors are
> > > > also done sending
> > > the packets.
> > > Yes.
> > >
> > > > 3) The check "if (txq->nb_tx_free < txq->tx_free_thresh)" is
> > > > mainly to ensure that we do not check the "descriptor done" status too
> often.
> > > This condition is to ensure there're enough free descriptors for Tx,
> > > avoid Tx ring full.
> > Ok. I think this check has another purpose as well, though I am not
> > sure if it is intentional. I see that the descriptors are initialized
> > with DTYPE set to 0xF (in function i40e_reset_tx_queue). So, in the
> > very first call to transmit function (for
> > ex: i40e_xmit_fixed_burst_vec), the 'i40e_tx_free_bufs' function would
> > end up checking the DTYPE field, if the above check was not there.
> >
> > In the data sheet, in section 8.4.2.1.1 (transmit data descriptor
> > format), the RS field is described as follows:
> >
> > " Report Status. When set, the hardware reports the DMA completion of
> > the transmit descriptor and its data buffer. Completion is reported by
> > descriptor write back or by head write back as configured by the
> > HEAD_WBEN flag in the transmit context. When it is reported by
> > descriptor write back, the DTYP field is set to 0xF and the RS flag is set."
> >
> > Considering the last sentence, should the code check for both DTYP
> > field and RS field for completion? Currently, the code checks for just
> > the DTYP field (in the function i40e_tx_free_bufs).
> 
> In my understanding, RS bit is set by SW to indicate HW needs to write back. So
> I don't think PMD needs to check RS bit.
> Besides, PMD should have ensured PMD checks DTYPE of the descriptor with RS
> bit only.
This is not ensured for the very first call to the transmit API. All the descriptors are initialized with DTYPE set to 0xF and the RS bit is set to 0. But the check "if (txq->nb_tx_free < txq->tx_free_thresh)" ensures we do not free these descriptors. 

> 
> >
> > >
> > > >
> > > > Is my understanding correct?
> > > >
> > > > Thank you,
> > > > Honnappa