RE: [PATCH v2] net/ixgbe: add proper memory barriers for some Rx functions

["Zhang, Qi Z" <qi.z.zhang@intel.com>]

> -----Original Message-----
> From: zhoumin <zhoumin@loongson.cn>
> Sent: Friday, May 5, 2023 10:43 AM
> To: Zhang, Qi Z <qi.z.zhang@intel.com>; Morten Brørup
> <mb@smartsharesystems.com>; Konstantin Ananyev
> <konstantin.v.ananyev@yandex.ru>
> Cc: dev@dpdk.org; maobibo@loongson.cn; Yang, Qiming
> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>;
> ruifeng.wang@arm.com; drc@linux.vnet.ibm.com; Tyler Retzlaff
> <roretzla@linux.microsoft.com>
> Subject: Re: [PATCH v2] net/ixgbe: add proper memory barriers for some Rx
> functions
> 
> Hi Qi,
> 
> On Thur, May 4, 2023 at 9:33PM, Zhang, Qi Z wrote:
> >
> >> -----Original Message-----
> >> From: Morten Brørup <mb@smartsharesystems.com>
> >> Sent: Thursday, May 4, 2023 9:22 PM
> >> To: zhoumin <zhoumin@loongson.cn>; Konstantin Ananyev
> >> <konstantin.v.ananyev@yandex.ru>
> >> Cc: dev@dpdk.org; maobibo@loongson.cn; Yang, Qiming
> >> <qiming.yang@intel.com>; Wu, Wenjun1 <wenjun1.wu@intel.com>;
> >> ruifeng.wang@arm.com; drc@linux.vnet.ibm.com; Tyler Retzlaff
> >> <roretzla@linux.microsoft.com>
> >> Subject: RE: [PATCH v2] net/ixgbe: add proper memory barriers for
> >> some Rx functions
> >>
> >>> From: zhoumin [mailto:zhoumin@loongson.cn]
> >>> Sent: Thursday, 4 May 2023 15.17
> >>>
> >>> Hi Konstantin,
> >>>
> >>> Thanks for your  comments.
> >>>
> >>> On 2023/5/1 下午9:29, Konstantin Ananyev wrote:
> >>>>> Segmentation fault has been observed while running the
> >>>>> ixgbe_recv_pkts_lro() function to receive packets on the Loongson
> >>>>> 3C5000 processor which has 64 cores and 4 NUMA nodes.
> >>>>>
> >>>>>  From the ixgbe_recv_pkts_lro() function, we found that as long as
> >>>>> the first packet has the EOP bit set, and the length of this
> >>>>> packet is less than or equal to rxq->crc_len, the segmentation
> >>>>> fault will definitely happen even though on the other platforms, such
> as X86.
> > Sorry to interrupt, but I am curious why this issue still exists on x86
> architecture. Can volatile be used to instruct the compiler to generate read
> instructions in a specific order, and does x86 guarantee not to reorder load
> operations?
> Actually, I did not see the segmentation fault on X86. I just made the first
> packet which had the EOP bit set had a zero length, then the segmentation
> fault would happen on X86. So, I thought that the out-of-order access to the
> descriptor might be possible to make the ready packet zero length, and this
> case was more likely to cause the segmentation fault.

I see, thanks for the explanation.

> >>>>> Because when processd the first packet the first_seg->next will be
> >>>>> NULL, if at the same time this packet has the EOP bit set and its
> >>>>> length is less than or equal to rxq->crc_len, the following loop
> >>>>> will be excecuted:
> >>>>>
> >>>>>      for (lp = first_seg; lp->next != rxm; lp = lp->next)
> >>>>>          ;
> >>>>>
> >>>>> We know that the first_seg->next will be NULL under this condition.
> >>>>> So the
> >>>>> expression of lp->next->next will cause the segmentation fault.
> >>>>>
> >>>>> Normally, the length of the first packet with EOP bit set will be
> >>>>> greater than rxq->crc_len. However, the out-of-order execution of
> >>>>> CPU may make the read ordering of the status and the rest of the
> >>>>> descriptor fields in this function not be correct. The related
> >>>>> codes are as following:
> >>>>>
> >>>>>          rxdp = &rx_ring[rx_id];
> >>>>>   #1     staterr = rte_le_to_cpu_32(rxdp->wb.upper.status_error);
> >>>>>
> >>>>>          if (!(staterr & IXGBE_RXDADV_STAT_DD))
> >>>>>              break;
> >>>>>
> >>>>>   #2     rxd = *rxdp;
> >>>>>
> >>>>> The sentence #2 may be executed before sentence #1. This action is
> >>>>> likely to make the ready packet zero length. If the packet is the
> >>>>> first packet and has the EOP bit set, the above segmentation fault
> >>>>> will happen.
> >>>>>
> >>>>> So, we should add rte_rmb() to ensure the read ordering be correct.
> >>>>> We also
> >>>>> did the same thing in the ixgbe_recv_pkts() function to make the
> >>>>> rxd data be valid even thougth we did not find segmentation fault
> >>>>> in this function.
> >>>>>
> >>>>> Signed-off-by: Min Zhou <zhoumin@loongson.cn>
> >>>>> ---
> >>>>> v2:
> >>>>> - Make the calling of rte_rmb() for all platforms
> >>>>> ---
> >>>>>   drivers/net/ixgbe/ixgbe_rxtx.c | 3 +++
> >>>>>   1 file changed, 3 insertions(+)
> >>>>>
> >>>>> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> >>>>> b/drivers/net/ixgbe/ixgbe_rxtx.c index c9d6ca9efe..302a5ab7ff
> >>>>> 100644
> >>>>> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> >>>>> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> >>>>> @@ -1823,6 +1823,8 @@ ixgbe_recv_pkts(void *rx_queue, struct
> >>>>> rte_mbuf **rx_pkts,
> >>>>>           staterr = rxdp->wb.upper.status_error;
> >>>>>           if (!(staterr & rte_cpu_to_le_32(IXGBE_RXDADV_STAT_DD)))
> >>>>>               break;
> >>>>> +
> >>>>> +        rte_rmb();
> >>>>>           rxd = *rxdp;
> >>>>
> >>>>
> >>>> Indeed, looks like a problem to me on systems with relaxed MO.
> >>>> Strange that it was never hit on arm or ppc - cc-ing ARM/PPC
> maintainers.
> >>> The LoongArch architecture uses the Weak Consistency model which can
> >>> cause the problem, especially in scenario with many cores, such as
> >>> Loongson 3C5000 with four NUMA node, which has 64 cores. I cannot
> >>> reproduce it on Loongson 3C5000 with one NUMA node, which just has
> >>> 16
> >> cores.
> >>>> About a fix - looks right, but a bit excessive to me - as I
> >>>> understand all we need here is to prevent re-ordering by CPU itself.
> >>> Yes, thanks for cc-ing.
> >>>> So rte_smp_rmb() seems enough here.
> >>>> Or might be just:
> >>>> staterr = __atomic_load_n(&rxdp->wb.upper.status_error,
> >>>> __ATOMIC_ACQUIRE);
> >>>>
> >>> Does __atomic_load_n() work on Windows if we use it to solve this
> >> problem ?
> >>
> >> Yes, __atomic_load_n() works on Windows too.
> >>
> Best regards,
> 
> Min
>