Re: [PATCH] examples/l3fwd: add option to set refetch offset

[Stephen Hemminger <stephen@networkplumber.org>]

On Tue, 14 Jan 2025 08:07:52 -0800
Stephen Hemminger <stephen@networkplumber.org> wrote:

> On Tue, 14 Jan 2025 17:22:08 +0800
> huangdengdui <huangdengdui@huawei.com> wrote:
> 
> > On 2025/1/11 1:20, Stephen Hemminger wrote:  
> > > This will make it slower for many platforms.
> > > GCC will unroll a loop of fixed small size, which is what we want.    
> > 
> > Do you mean to replace option with a macro?
> > But most of prefetch_offset are used with the nb_rx, So using macros is the same as using options.
> > 
> > const int32_t k = RTE_ALIGN_FLOOR(nb_rx, FWDSTEP);
> > for (j = 0; j != k; j += FWDSTEP) {
> > 	for (i = 0, pos = j + prefetch_offset;
> > 	     i < FWDSTEP && pos < k; i++, pos++)
> > 		rte_prefetch0(rte_pktmbuf_mtod(pkts_burst[pos], void *));
> > 	processx4_step1(&pkts_burst[j], &dip, &ipv4_flag);
> > 	processx4_step2(qconf, dip, ipv4_flag, portid,
> > 			&pkts_burst[j], &dst_port[j]);
> > 	if (do_step3)
> > 		processx4_step3(&pkts_burst[j], &dst_port[j]);
> > }
> > 
> > The option can dynamically adjust the prefetch window, which makes it easier to find the prefetch window for a HW platform.
> > So I think it's better to use option.  
> 
> The tradeoff is that loop unrolling most often is only done on small fix sized loops.
> And the cost of a loop with variable small values (branch prediction) is high enough that it 
> could make things slower.
> 
> Prefetching is a balancing act, and more is not better especially on real workloads.

You might also want to look at the quad loop model used in VPP for prefetching.

https://my-vpp-docs.readthedocs.io/en/latest/gettingstarted/developers/vnet.html