[PATCH] net/qede: reduce the optimization level

[PATCH] net/qede: reduce the optimization level

[Thierry Herbelot]

The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
compiled with -O3 (default level for all DPDK code). A bug is opened
for this issue (see Link).

A first workaround is to just disable all optimizations (-O0), which
restores packet Rx with Qlogic NICs. However, the performance impact
is not acceptable (around 50% drop).

A better compromise is to use -O1 for the qede PMD:
- there is some perf impact,
- but the PMD is working as expected (packets are correctly received).

When compiling with both -O2 and -O3 and gcc-13, there is no packet
reception when using the qede PMD. The root cause could be missing
'volatile' keywords or missing memory barriers in the qede PMD code.

Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>
---
 drivers/net/qede/base/meson.build | 1 +
 drivers/net/qede/meson.build      | 1 +
 2 files changed, 2 insertions(+)

diff --git a/drivers/net/qede/base/meson.build b/drivers/net/qede/base/meson.build
index a6dad3ec7bcc..6c8800a3f48f 100644
--- a/drivers/net/qede/base/meson.build
+++ b/drivers/net/qede/base/meson.build
@@ -55,3 +55,4 @@ foreach flag: error_cflags
         base_cflags += flag
     endif
 endforeach
+base_cflags += '-O1'
diff --git a/drivers/net/qede/meson.build b/drivers/net/qede/meson.build
index e1b21d6ff5cd..3c2a5205732a 100644
--- a/drivers/net/qede/meson.build
+++ b/drivers/net/qede/meson.build
@@ -22,3 +22,4 @@ sources = files(
 if cc.has_argument('-Wno-format-nonliteral')
     cflags += '-Wno-format-nonliteral'
 endif
+cflags += '-O1'
-- 
2.39.2

Re: [PATCH] net/qede: reduce the optimization level

[Stephen Hemminger]

On Wed,  3 Sep 2025 13:32:52 +0200
Thierry Herbelot <thierry.herbelot@6wind.com> wrote:

> The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
> compiled with -O3 (default level for all DPDK code). A bug is opened
> for this issue (see Link).
> 
> A first workaround is to just disable all optimizations (-O0), which
> restores packet Rx with Qlogic NICs. However, the performance impact
> is not acceptable (around 50% drop).
> 
> A better compromise is to use -O1 for the qede PMD:
> - there is some perf impact,
> - but the PMD is working as expected (packets are correctly received).
> 
> When compiling with both -O2 and -O3 and gcc-13, there is no packet
> reception when using the qede PMD. The root cause could be missing
> 'volatile' keywords or missing memory barriers in the qede PMD code.
> 
> Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
> Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>

Since GCC 14 and 15 are available do they fix the problem?
If so, the the hack should be limited to Gcc 13 and with
more complete comment.

[V2] net/qede: reduce the optimization level for gcc > 11

[Thierry Herbelot]

The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
compiled with -O3 (default level for all DPDK code). A bug is opened
for this issue (see Link). The same issue is also seen with gcc-12
and gcc-14.

A first workaround is to just disable all optimizations (-O0), which
restores packet Rx with Qlogic NICs. However, the performance impact
is not acceptable (around 50% drop).

A better compromise is to use -O1 for the qede PMD:
- there is some perf impact,
- but the PMD is working as expected (packets are correctly received).

When compiling with both -O2 and -O3 and gcc > 11, there is no packet
reception when using the qede PMD. The root cause could be missing
'volatile' keywords or missing memory barriers in the qede PMD code.

Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>
---
V2: be more precise when dropping optimization level (only for gcc > 11)
---
 drivers/net/qede/base/meson.build | 4 ++++
 drivers/net/qede/meson.build      | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/net/qede/base/meson.build b/drivers/net/qede/base/meson.build
index a6dad3ec7bcc..130835cc31b1 100644
--- a/drivers/net/qede/base/meson.build
+++ b/drivers/net/qede/base/meson.build
@@ -55,3 +55,7 @@ foreach flag: error_cflags
         base_cflags += flag
     endif
 endforeach
+# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
+if (cc.get_id() == 'gcc' and cc.version().version_compare('>11.0'))
+    base_cflags += '-O1'
+endif
diff --git a/drivers/net/qede/meson.build b/drivers/net/qede/meson.build
index e1b21d6ff5cd..e4268883ff93 100644
--- a/drivers/net/qede/meson.build
+++ b/drivers/net/qede/meson.build
@@ -22,3 +22,7 @@ sources = files(
 if cc.has_argument('-Wno-format-nonliteral')
     cflags += '-Wno-format-nonliteral'
 endif
+# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
+if (cc.get_id() == 'gcc' and cc.version().version_compare('>11.0'))
+    cflags += '-O1'
+endif
-- 
2.39.2

Re: [V2] net/qede: reduce the optimization level for gcc > 11

[Stephen Hemminger]

On Mon,  8 Sep 2025 09:04:13 +0200
Thierry Herbelot <thierry.herbelot@6wind.com> wrote:

> +# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
> +if (cc.get_id() == 'gcc' and cc.version().version_compare('>11.0'))
> +    base_cflags += '-O1'
> +endif

Did you test with Gcc 14 and 15, it may be the bug is already fixed.

Re: [V2] net/qede: reduce the optimization level for gcc > 11

[Thierry Herbelot]

On 9/8/25 21:59, Stephen Hemminger wrote:
> On Mon,  8 Sep 2025 09:04:13 +0200
> Thierry Herbelot <thierry.herbelot@6wind.com> wrote:
> 
>> +# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
>> +if (cc.get_id() == 'gcc' and cc.version().version_compare('>11.0'))
>> +    base_cflags += '-O1'
>> +endif
> 
> Did you test with Gcc 14 and 15, it may be the bug is already fixed.

As said in the commit log:

"The same issue is also seen with gcc-12 and gcc-14."

I did not test with gcc-15.

The issue is not seen with clang20 (qede PMD compiled with -O3 works as 
expected).

	Best regards

	Thierry

-- 
Thierry Herbelot
CSE - Titulaire au collège Cadres
Senior Software Engineer
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

[V3] net/qede: reduce the optimization level for gcc > 11

[Thierry Herbelot]

The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
compiled with -O3 (default level for all DPDK code). A bug is opened
for this issue (see Link). The same issue is also seen with gcc-12
and gcc-14. The issue is not seen with clang-20.

A first workaround is to just disable all optimizations (-O0), which
restores packet Rx with Qlogic NICs. However, the performance impact
is not acceptable (around 50% drop).

A better compromise is to use -O1 for the qede PMD:
- there is some perf impact,
- but the PMD is working as expected (packets are correctly received).

When compiling with both -O2 and -O3 and gcc > 11, there is no packet
reception when using the qede PMD. The root cause could be missing
'volatile' keywords or missing memory barriers in the qede PMD code.

Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>
---
V2: be more precise when dropping optimization level (only for gcc > 11)
V3: even more precise: the issue appears with gcc-12
---
 drivers/net/qede/base/meson.build | 4 ++++
 drivers/net/qede/meson.build      | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/net/qede/base/meson.build b/drivers/net/qede/base/meson.build
index a6dad3ec7bcc..acab6e8ac938 100644
--- a/drivers/net/qede/base/meson.build
+++ b/drivers/net/qede/base/meson.build
@@ -55,3 +55,7 @@ foreach flag: error_cflags
         base_cflags += flag
     endif
 endforeach
+# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
+if (cc.get_id() == 'gcc' and cc.version().version_compare('>=12.0'))
+    base_cflags += '-O1'
+endif
diff --git a/drivers/net/qede/meson.build b/drivers/net/qede/meson.build
index e1b21d6ff5cd..86c3503a82f6 100644
--- a/drivers/net/qede/meson.build
+++ b/drivers/net/qede/meson.build
@@ -22,3 +22,7 @@ sources = files(
 if cc.has_argument('-Wno-format-nonliteral')
     cflags += '-Wno-format-nonliteral'
 endif
+# no packet Rx with gcc > 11 with compiling with default -O3 or -O2
+if (cc.get_id() == 'gcc' and cc.version().version_compare('>=12.0'))
+    cflags += '-O1'
+endif
-- 
2.39.2

Re: [V3] net/qede: reduce the optimization level for gcc > 11

[Stephen Hemminger]

On Tue,  9 Sep 2025 07:40:23 +0200
Thierry Herbelot <thierry.herbelot@6wind.com> wrote:

> The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
> compiled with -O3 (default level for all DPDK code). A bug is opened
> for this issue (see Link). The same issue is also seen with gcc-12
> and gcc-14. The issue is not seen with clang-20.
> 
> A first workaround is to just disable all optimizations (-O0), which
> restores packet Rx with Qlogic NICs. However, the performance impact
> is not acceptable (around 50% drop).
> 
> A better compromise is to use -O1 for the qede PMD:
> - there is some perf impact,
> - but the PMD is working as expected (packets are correctly received).
> 
> When compiling with both -O2 and -O3 and gcc > 11, there is no packet
> reception when using the qede PMD. The root cause could be missing
> 'volatile' keywords or missing memory barriers in the qede PMD code.
> 
> Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
> Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>

Which rx_burst gets used on that hardware?

Is there any indication of errors (like rx_mbuf_alloc_failed) in the stats.

The driver does not appear to have any write barrier after updating the consumed packets.

Would this help?
diff --git a/drivers/net/qede/qede_rxtx.c b/drivers/net/qede/qede_rxtx.c
index 25e28fd9f6..1b5109d966 100644
--- a/drivers/net/qede/qede_rxtx.c
+++ b/drivers/net/qede/qede_rxtx.c
@@ -1295,6 +1295,8 @@ static inline void qede_rx_bd_ring_consume(struct qede_rx_queue *rxq)
 {
        ecore_chain_consume(&rxq->rx_bd_ring);
        rxq->sw_rx_cons++;
+
+       rte_wmb();
 }
 

Re: [V3] net/qede: reduce the optimization level for gcc > 11

[Thierry Herbelot]

Hello Stephen,

On 9/9/25 18:17, Stephen Hemminger wrote:
> On Tue,  9 Sep 2025 07:40:23 +0200
> Thierry Herbelot <thierry.herbelot@6wind.com> wrote:
> 
>> The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
>> compiled with -O3 (default level for all DPDK code). A bug is opened
>> for this issue (see Link). The same issue is also seen with gcc-12
>> and gcc-14. The issue is not seen with clang-20.
>>
>> A first workaround is to just disable all optimizations (-O0), which
>> restores packet Rx with Qlogic NICs. However, the performance impact
>> is not acceptable (around 50% drop).
>>
>> A better compromise is to use -O1 for the qede PMD:
>> - there is some perf impact,
>> - but the PMD is working as expected (packets are correctly received).
>>
>> When compiling with both -O2 and -O3 and gcc > 11, there is no packet
>> reception when using the qede PMD. The root cause could be missing
>> 'volatile' keywords or missing memory barriers in the qede PMD code.
>>
>> Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
>> Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>
> 
> Which rx_burst gets used on that hardware?
> 
> Is there any indication of errors (like rx_mbuf_alloc_failed) in the stats.
> 
> The driver does not appear to have any write barrier after updating the consumed packets.
> 
> Would this help?
> diff --git a/drivers/net/qede/qede_rxtx.c b/drivers/net/qede/qede_rxtx.c
> index 25e28fd9f6..1b5109d966 100644
> --- a/drivers/net/qede/qede_rxtx.c
> +++ b/drivers/net/qede/qede_rxtx.c
> @@ -1295,6 +1295,8 @@ static inline void qede_rx_bd_ring_consume(struct qede_rx_queue *rxq)
>   {
>          ecore_chain_consume(&rxq->rx_bd_ring);
>          rxq->sw_rx_cons++;
> +
> +       rte_wmb();
>   }
>   
I just checked with this patch, but the error is still present:

testpmd> show fwd stats all

   ---------------------- Forward statistics for port 0 
----------------------
   RX-packets: 0              RX-dropped: 10            RX-total: 10
   RX-error: 0
   RX-nombufs: 10
   TX-packets: 0              TX-dropped: 0             TX-total: 0
  
----------------------------------------------------------------------------

The error chain is:

RX-nombufs
  => (testpmd) stats.rx_nombuf
   => (qede) p_stats->common.no_buff_discards
    => (qede/base) HILO_64_REGPAIR(mstats.no_buff_discard)
          (internal to the NIC)

	Best regards

	Thierry


> 
> 
> 

-- 
Thierry Herbelot

Re: [V3] net/qede: reduce the optimization level for gcc > 11

[Stephen Hemminger]

On Wed, 10 Sep 2025 08:49:38 +0200
Thierry Herbelot <thierry.herbelot@6wind.com> wrote:

> Hello Stephen,
> 
> On 9/9/25 18:17, Stephen Hemminger wrote:
> > On Tue,  9 Sep 2025 07:40:23 +0200
> > Thierry Herbelot <thierry.herbelot@6wind.com> wrote:
> >   
> >> The qede PMD stopped working under Ubuntu-24.04 (using gcc-13) when
> >> compiled with -O3 (default level for all DPDK code). A bug is opened
> >> for this issue (see Link). The same issue is also seen with gcc-12
> >> and gcc-14. The issue is not seen with clang-20.
> >>
> >> A first workaround is to just disable all optimizations (-O0), which
> >> restores packet Rx with Qlogic NICs. However, the performance impact
> >> is not acceptable (around 50% drop).
> >>
> >> A better compromise is to use -O1 for the qede PMD:
> >> - there is some perf impact,
> >> - but the PMD is working as expected (packets are correctly received).
> >>
> >> When compiling with both -O2 and -O3 and gcc > 11, there is no packet
> >> reception when using the qede PMD. The root cause could be missing
> >> 'volatile' keywords or missing memory barriers in the qede PMD code.
> >>
> >> Link: https://bugs.dpdk.org/show_bug.cgi?id=1379
> >> Signed-off-by: Thierry Herbelot <thierry.herbelot@6wind.com>  
> > 
> > Which rx_burst gets used on that hardware?
> > 
> > Is there any indication of errors (like rx_mbuf_alloc_failed) in the stats.
> > 
> > The driver does not appear to have any write barrier after updating the consumed packets.
> > 
> > Would this help?
> > diff --git a/drivers/net/qede/qede_rxtx.c b/drivers/net/qede/qede_rxtx.c
> > index 25e28fd9f6..1b5109d966 100644
> > --- a/drivers/net/qede/qede_rxtx.c
> > +++ b/drivers/net/qede/qede_rxtx.c
> > @@ -1295,6 +1295,8 @@ static inline void qede_rx_bd_ring_consume(struct qede_rx_queue *rxq)
> >   {
> >          ecore_chain_consume(&rxq->rx_bd_ring);
> >          rxq->sw_rx_cons++;
> > +
> > +       rte_wmb();
> >   }
> >     
> I just checked with this patch, but the error is still present:
> 
> testpmd> show fwd stats all  
> 
>    ---------------------- Forward statistics for port 0 
> ----------------------
>    RX-packets: 0              RX-dropped: 10            RX-total: 10
>    RX-error: 0
>    RX-nombufs: 10
>    TX-packets: 0              TX-dropped: 0             TX-total: 0
>   
> ----------------------------------------------------------------------------
> 
> The error chain is:
> 
> RX-nombufs
>   => (testpmd) stats.rx_nombuf
>    => (qede) p_stats->common.no_buff_discards
>     => (qede/base) HILO_64_REGPAIR(mstats.no_buff_discard)  
>           (internal to the NIC)
> 

Thanks, lets root cause this (rather than paper it over) because
other drivers may have same issue. Since no mbufs are present
looks like rx discriptor never made it to NIC

Re: [V3] net/qede: reduce the optimization level for gcc > 11

[David Marchand]

On Wed, 10 Sept 2025 at 18:42, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Wed, 10 Sep 2025 08:49:38 +0200
> Thierry Herbelot <thierry.herbelot@6wind.com> wrote:
> > The error chain is:
> >
> > RX-nombufs
> >   => (testpmd) stats.rx_nombuf
> >    => (qede) p_stats->common.no_buff_discards
> >     => (qede/base) HILO_64_REGPAIR(mstats.no_buff_discard)
> >           (internal to the NIC)
> >
>
> Thanks, lets root cause this (rather than paper it over) because

+1

> other drivers may have same issue. Since no mbufs are present
> looks like rx discriptor never made it to NIC

I would add that if the descs do make it to the nic, maybe the (IOVA)
addresses are incorrect.
We changed the default IOVA not so long ago.. so maybe play with
--iova-mode to continue investigations.


-- 
David Marchand