Ring library optimization idea

Ring library optimization idea

[Morten Brørup]

Most of the fast path ring library functions access the two cache lines containing respectively the r->prod and r->cons structures.

Some of the fast path functions also have to fetch r->capacity and r->mask, which reside in another cache line.

What do you think about adding shadow variables of r->capacity and r->mask to the same cache line as r->prod, to improve ring library performance (by avoiding having to read a third cache line) when under CPU cache pressure?

-Morten

RE: Ring library optimization idea

[Honnappa Nagarahalli]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Sunday, August 20, 2023 4:24 AM
> To: dev@dpdk.org
> Cc: Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>;
> konstantin.v.ananyev@yandex.ru
> Subject: Ring library optimization idea
> 
> Most of the fast path ring library functions access the two cache lines containing
> respectively the r->prod and r->cons structures.
> 
> Some of the fast path functions also have to fetch r->capacity and r->mask,
> which reside in another cache line.
> 
> What do you think about adding shadow variables of r->capacity and r->mask
> to the same cache line as r->prod, to improve ring library performance (by
> avoiding having to read a third cache line) when under CPU cache pressure?
Agree with the idea. I am not sure about the performance as these are read-only data. IMO, we should not worry about it as it saves one cache line (but we do not need to measure the performance). They will occupy 12B, but I do not see how we will occupy the entire 64B of the cacheline in the near future.

Also, the shadow variables need to be part of both r->prod and r->cons.

> 
> -Morten

RE: Ring library optimization idea

[Konstantin Ananyev]

Hi Morten,

> > Most of the fast path ring library functions access the two cache lines containing
> > respectively the r->prod and r->cons structures.
> >
> > Some of the fast path functions also have to fetch r->capacity and r->mask,
> > which reside in another cache line.
> >
> > What do you think about adding shadow variables of r->capacity and r->mask
> > to the same cache line as r->prod, to improve ring library performance (by
> > avoiding having to read a third cache line) when under CPU cache pressure?
> Agree with the idea. I am not sure about the performance as these are read-only data. IMO, we should not worry about it as it saves
> one cache line (but we do not need to measure the performance). They will occupy 12B, but I do not see how we will occupy the
> entire 64B of the cacheline in the near future.
> 
> Also, the shadow variables need to be part of both r->prod and r->cons.

Personally, I think that for majority of cases the perf gain with be either tiny or none.
From other side, I don't see any harm with that approach, as Honnapa said, we
do should have space for that in prod/cons cache-lines.
So, no objections from me in general.
Thanks
Konstantin