[dpdk-dev] Minimun value of RTE_MAX_LCORE

[dpdk-dev] Minimun value of RTE_MAX_LCORE

[Juraj Linkeš]

Hi dpdk devs,

Is there a constraint on how low RTE_MAX_LCORE can be? I'm implementing a discovery mechanism that sets RTE_MAX_LCORE according to the number of host cores, but I'm hitting errors when the values are low:
https://travis-ci.com/github/jlinkes/dpdk/jobs/399596828
Message: Found 2
cores
Message: Found 1
numa nodes

../app/test/test_rcu_qsbr.c:296:54: error: iteration 2 invokes undefined behavior [-Werror=aggressive-loop-optimizations]

../app/test/test_rcu_qsbr.c:315:55: error: array subscript is above array bounds [-Werror=array-bounds]

All VM jobs failed in that Travis build. Travis VMs only have 2 cores, so I tried to put a bound on the build. I set it to 4 and all jobs except GCC shared lib jobs passed, which still threw iteration 4 invokes undefined behavior error:
https://travis-ci.com/github/jlinkes/dpdk/jobs/400004089

../examples/performance-thread/l3fwd-thread/main.c:2338:34: error: iteration 4 invokes undefined behavior [-Werror=aggressive-loop-optimizations]

This happens for number of cores < 32 and looks like a limitation unique to l3fwd (with cores between 4 and 32 - I didn't see the error elsewhere).

Should I use the bound or are these legitimate errors? The fact that only GCC (and not clang) shared lib jobs failed is also suspicious.

Thanks,
Juraj

Re: [dpdk-dev] Minimun value of RTE_MAX_LCORE

[Bruce Richardson]

On Thu, Oct 15, 2020 at 10:49:17AM +0000, Juraj Linkeš wrote:
> Hi dpdk devs,
> 
> Is there a constraint on how low RTE_MAX_LCORE can be? I'm implementing a
> discovery mechanism that sets RTE_MAX_LCORE according to the number of
> host cores, but I'm hitting errors when the values are low:
> https://travis-ci.com/github/jlinkes/dpdk/jobs/399596828 Message: Found 2
> cores Message: Found 1 numa nodes
> 
> ../app/test/test_rcu_qsbr.c:296:54: error: iteration 2 invokes undefined
> behavior [-Werror=aggressive-loop-optimizations]
> 
> ../app/test/test_rcu_qsbr.c:315:55: error: array subscript is above array
> bounds [-Werror=array-bounds]
> 
> All VM jobs failed in that Travis build. Travis VMs only have 2 cores, so
> I tried to put a bound on the build. I set it to 4 and all jobs except
> GCC shared lib jobs passed, which still threw iteration 4 invokes
> undefined behavior error:
> https://travis-ci.com/github/jlinkes/dpdk/jobs/400004089
> 
> ../examples/performance-thread/l3fwd-thread/main.c:2338:34: error:
> iteration 4 invokes undefined behavior
> [-Werror=aggressive-loop-optimizations]
> 
> This happens for number of cores < 32 and looks like a limitation unique
> to l3fwd (with cores between 4 and 32 - I didn't see the error
> elsewhere).
> 
> Should I use the bound or are these legitimate errors? The fact that only
> GCC (and not clang) shared lib jobs failed is also suspicious.
>
Four seems a reasonable lower bound to have, but 32 seems high. I agree
that the warning itself looks suspicious too.

However, looking at the code and the line in question, I think there is
indeed an app bug.  The loop goes up to MAX_ETHPORTS, but the tx_bufs array
being dereferenced is dimensioned by MAX_LCORES, which is why you hit
problems with lcore values below 32 (==MAX_ETHPORTS). Based on the use, I
suspect the structure definition may be wrong and the array should be
dimensioned for MAX_ETHPORTS - but I haven't dug into the code further than
that.

Regards,
/Bruce

Re: [dpdk-dev] Minimun value of RTE_MAX_LCORE

[Gaëtan Rivet]

On 15/10/20 10:49 +0000, Juraj Linkeš wrote:
> Hi dpdk devs,
> 
> Is there a constraint on how low RTE_MAX_LCORE can be? I'm implementing a discovery mechanism that sets RTE_MAX_LCORE according to the number of host cores, but I'm hitting errors when the values are low:
> https://travis-ci.com/github/jlinkes/dpdk/jobs/399596828
> Message: Found 2
> cores
> Message: Found 1
> numa nodes
> 
> ../app/test/test_rcu_qsbr.c:296:54: error: iteration 2 invokes undefined behavior [-Werror=aggressive-loop-optimizations]
> 
> ../app/test/test_rcu_qsbr.c:315:55: error: array subscript is above array bounds [-Werror=array-bounds]
> 
> All VM jobs failed in that Travis build. Travis VMs only have 2 cores, so I tried to put a bound on the build. I set it to 4 and all jobs except GCC shared lib jobs passed, which still threw iteration 4 invokes undefined behavior error:
> https://travis-ci.com/github/jlinkes/dpdk/jobs/400004089
> 
> ../examples/performance-thread/l3fwd-thread/main.c:2338:34: error: iteration 4 invokes undefined behavior [-Werror=aggressive-loop-optimizations]
> 
> This happens for number of cores < 32 and looks like a limitation unique to l3fwd (with cores between 4 and 32 - I didn't see the error elsewhere).
> 
> Should I use the bound or are these legitimate errors? The fact that only GCC (and not clang) shared lib jobs failed is also suspicious.
> 
> Thanks,
> Juraj
> 

Hi,

I can see a CPU config setting it to 4, so it might be a valid value.
Not sure it would be the lower bound though.

However, I think the issue you get here shows why your discovery
mechanism is not great.  Most of the time, DPDK applications are not
built on their target machine: either due to CI (like your issue),
automatic packaging, cross-compilation for smartNIC, etc.

Platforms that would benefit from your discovery mechanism will define
RTE_MAX_LCORE explicitly, e.g. in config/arm/meson.build, line 34 and
further.

Regards,
-- 
Gaëtan

Re: [dpdk-dev] Minimun value of RTE_MAX_LCORE

[Juraj Linkeš]

> -----Original Message-----
> From: Gaëtan Rivet <grive@u256.net>
> Sent: Thursday, October 15, 2020 1:31 PM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Minimun value of RTE_MAX_LCORE
> 
> On 15/10/20 10:49 +0000, Juraj Linkeš wrote:
> > Hi dpdk devs,
> >
> > Is there a constraint on how low RTE_MAX_LCORE can be? I'm implementing a
> discovery mechanism that sets RTE_MAX_LCORE according to the number of
> host cores, but I'm hitting errors when the values are low:
> > https://travis-ci.com/github/jlinkes/dpdk/jobs/399596828
> > Message: Found 2
> > cores
> > Message: Found 1
> > numa nodes
> >
> > ../app/test/test_rcu_qsbr.c:296:54: error: iteration 2 invokes
> > undefined behavior [-Werror=aggressive-loop-optimizations]
> >
> > ../app/test/test_rcu_qsbr.c:315:55: error: array subscript is above
> > array bounds [-Werror=array-bounds]
> >
> > All VM jobs failed in that Travis build. Travis VMs only have 2 cores, so I tried to
> put a bound on the build. I set it to 4 and all jobs except GCC shared lib jobs
> passed, which still threw iteration 4 invokes undefined behavior error:
> > https://travis-ci.com/github/jlinkes/dpdk/jobs/400004089
> >
> > ../examples/performance-thread/l3fwd-thread/main.c:2338:34: error:
> > iteration 4 invokes undefined behavior
> > [-Werror=aggressive-loop-optimizations]
> >
> > This happens for number of cores < 32 and looks like a limitation unique to
> l3fwd (with cores between 4 and 32 - I didn't see the error elsewhere).
> >
> > Should I use the bound or are these legitimate errors? The fact that only GCC
> (and not clang) shared lib jobs failed is also suspicious.
> >
> > Thanks,
> > Juraj
> >
> 
> Hi,
> 
> I can see a CPU config setting it to 4, so it might be a valid value.
> Not sure it would be the lower bound though.
> 
> However, I think the issue you get here shows why your discovery mechanism is
> not great.  Most of the time, DPDK applications are not built on their target
> machine: either due to CI (like your issue), automatic packaging, cross-
> compilation for smartNIC, etc.
> 

It's not supposed to be great, just be a more sensible default for native builds. Users can still specify the cores on the command line if they like. Or do the default build (which will use predefined values).

> Platforms that would benefit from your discovery mechanism will define
> RTE_MAX_LCORE explicitly, e.g. in config/arm/meson.build, line 34 and further.
> 

I'm not sure what you mean. If they define it explicitly, then they can't benefit from the discoreved values. What we're exploring is using the discovered values for native builds and moving the statically defined RTE_MAX_LCORE and RTE_MAX_NUMA_NODES to cross files. Then we'll have native builds which better match the build machine and if anyone wants the target to be a particular SoC, they can use a cross file.

> Regards,
> --
> Gaëtan

Re: [dpdk-dev] Minimun value of RTE_MAX_LCORE

[Honnappa Nagarahalli]

<snip>

> >
> > On 15/10/20 10:49 +0000, Juraj Linkeš wrote:
> > > Hi dpdk devs,
> > >
> > > Is there a constraint on how low RTE_MAX_LCORE can be? I'm
> > > implementing a
> > discovery mechanism that sets RTE_MAX_LCORE according to the number
> of
> > host cores, but I'm hitting errors when the values are low:
> > > https://travis-ci.com/github/jlinkes/dpdk/jobs/399596828
> > > Message: Found 2
> > > cores
> > > Message: Found 1
> > > numa nodes
> > >
> > > ../app/test/test_rcu_qsbr.c:296:54: error: iteration 2 invokes
> > > undefined behavior [-Werror=aggressive-loop-optimizations]
> > >
> > > ../app/test/test_rcu_qsbr.c:315:55: error: array subscript is above
> > > array bounds [-Werror=array-bounds]
> > >
> > > All VM jobs failed in that Travis build. Travis VMs only have 2
> > > cores, so I tried to
> > put a bound on the build. I set it to 4 and all jobs except GCC shared
> > lib jobs passed, which still threw iteration 4 invokes undefined behavior
> error:
> > > https://travis-ci.com/github/jlinkes/dpdk/jobs/400004089
> > >
> > > ../examples/performance-thread/l3fwd-thread/main.c:2338:34: error:
> > > iteration 4 invokes undefined behavior
> > > [-Werror=aggressive-loop-optimizations]
> > >
> > > This happens for number of cores < 32 and looks like a limitation
> > > unique to
> > l3fwd (with cores between 4 and 32 - I didn't see the error elsewhere).
> > >
> > > Should I use the bound or are these legitimate errors? The fact that
> > > only GCC
> > (and not clang) shared lib jobs failed is also suspicious.
2 would be the minimum (one for main and one for worker), not less than that. Please check [1], I have fixed both of them.

[1] https://patches.dpdk.org/patch/81028/

> > >
> > > Thanks,
> > > Juraj
> > >
> >
> > Hi,
> >
> > I can see a CPU config setting it to 4, so it might be a valid value.
> > Not sure it would be the lower bound though.
> >
> > However, I think the issue you get here shows why your discovery
> > mechanism is not great.  Most of the time, DPDK applications are not
> > built on their target
> > machine: either due to CI (like your issue), automatic packaging,
> > cross- compilation for smartNIC, etc.
> >
> 
> It's not supposed to be great, just be a more sensible default for native
> builds. Users can still specify the cores on the command line if they like. Or do
> the default build (which will use predefined values).
> 
> > Platforms that would benefit from your discovery mechanism will define
> > RTE_MAX_LCORE explicitly, e.g. in config/arm/meson.build, line 34 and
> further.
> >
> 
> I'm not sure what you mean. If they define it explicitly, then they can't
> benefit from the discoreved values. What we're exploring is using the
> discovered values for native builds and moving the statically defined
> RTE_MAX_LCORE and RTE_MAX_NUMA_NODES to cross files. Then we'll
> have native builds which better match the build machine and if anyone wants
> the target to be a particular SoC, they can use a cross file.
Agree. I think there is enough flexibility provided for different use cases.

> 
> > Regards,
> > --
> > Gaëtan