[dpdk-dev] Limiting packet buffers under 4GB

[dpdk-dev] Limiting packet buffers under 4GB

[Jerin Jacob Kollanukkaran]

Hi Anatoly and All,

Just wondering what would the side effect of lowering a  _bit_ of static uint64_t baseaddr = 0x100000000 in 
lib/librte_eal/common/eal_common_memory.c for 64bit systems.

Use case:
If we _reserve_ VA address which less than 2^32 ONLY for packet buffers(mbuf), The use cases like
Pipeline, where need to transfer packets from one core to another cores can use ring element
size of 4B(32bit) which will reduce the a lot of read and write  to enable better
performance.

i.e Since upper 32bits will be zero, it is matter of typecasting of item to read and write from/to ring.
Essentially memcpy overhead for moving pointers over the ring will be half.

Is baseaddr set to 2^32 to make sure that secondary process will have more _chance_ of getting
the baseaddr in order for DPDK to work?

Thoughts on above? On general to reduce the mbuf pointer storage requirement for ring?



 

Re: [dpdk-dev] Limiting packet buffers under 4GB

[Olivier Matz]

Hi Jerin,

On Mon, Nov 04, 2019 at 12:59:40PM +0000, Jerin Jacob Kollanukkaran wrote:
> Hi Anatoly and All,
> 
> Just wondering what would the side effect of lowering a  _bit_ of static uint64_t baseaddr = 0x100000000 in 
> lib/librte_eal/common/eal_common_memory.c for 64bit systems.
> 
> Use case:
> If we _reserve_ VA address which less than 2^32 ONLY for packet buffers(mbuf), The use cases like
> Pipeline, where need to transfer packets from one core to another cores can use ring element
> size of 4B(32bit) which will reduce the a lot of read and write  to enable better
> performance.
> 
> i.e Since upper 32bits will be zero, it is matter of typecasting of item to read and write from/to ring.
> Essentially memcpy overhead for moving pointers over the ring will be half.
> 
> Is baseaddr set to 2^32 to make sure that secondary process will have more _chance_ of getting
> the baseaddr in order for DPDK to work?
> 
> Thoughts on above? On general to reduce the mbuf pointer storage requirement for ring?

There are use cases where the mbufs take more than 4GB of memory,
typically for protocols that do a lot of buffering/queing. So if
anything is done in that direction, it has to be an option.

Re: [dpdk-dev] Limiting packet buffers under 4GB

[Burakov, Anatoly]

On 04-Nov-19 12:59 PM, Jerin Jacob Kollanukkaran wrote:
> Hi Anatoly and All,
> 
> Just wondering what would the side effect of lowering a  _bit_ of static uint64_t baseaddr = 0x100000000 in
> lib/librte_eal/common/eal_common_memory.c for 64bit systems.
> 
> Use case:
> If we _reserve_ VA address which less than 2^32 ONLY for packet buffers(mbuf), The use cases like
> Pipeline, where need to transfer packets from one core to another cores can use ring element
> size of 4B(32bit) which will reduce the a lot of read and write  to enable better
> performance.
> 
> i.e Since upper 32bits will be zero, it is matter of typecasting of item to read and write from/to ring.
> Essentially memcpy overhead for moving pointers over the ring will be half.
> 
> Is baseaddr set to 2^32 to make sure that secondary process will have more _chance_ of getting
> the baseaddr in order for DPDK to work?
> 
> Thoughts on above? On general to reduce the mbuf pointer storage requirement for ring?
> 

You can already try that with --base-virtaddr option if you have such 
specific addressing requirements. That said, the address is pretty much 
arbitrary, so i'm not against lowering it in principle, it just feels 
like a workaround for something that's very specific to the workloads 
you are targetting.

-- 
Thanks,
Anatoly

Re: [dpdk-dev] Limiting packet buffers under 4GB

[Jerin Jacob]

On Wed, 6 Nov, 2019, 5:46 pm Burakov, Anatoly, <anatoly.burakov@intel.com>
wrote:

> On 04-Nov-19 12:59 PM, Jerin Jacob Kollanukkaran wrote:
> > Hi Anatoly and All,
> >
> > Just wondering what would the side effect of lowering a  _bit_ of static
> uint64_t baseaddr = 0x100000000 in
> > lib/librte_eal/common/eal_common_memory.c for 64bit systems.
> >
> > Use case:
> > If we _reserve_ VA address which less than 2^32 ONLY for packet
> buffers(mbuf), The use cases like
> > Pipeline, where need to transfer packets from one core to another cores
> can use ring element
> > size of 4B(32bit) which will reduce the a lot of read and write  to
> enable better
> > performance.
> >
> > i.e Since upper 32bits will be zero, it is matter of typecasting of item
> to read and write from/to ring.
> > Essentially memcpy overhead for moving pointers over the ring will be
> half.
> >
> > Is baseaddr set to 2^32 to make sure that secondary process will have
> more _chance_ of getting
> > the baseaddr in order for DPDK to work?
> >
> > Thoughts on above? On general to reduce the mbuf pointer storage
> requirement for ring?
> >
>
> You can already try that with --base-virtaddr option if you have such
> specific addressing requirements. That said, the address is pretty much
> arbitrary, so i'm not against lowering it in principle, it just feels
> like a workaround for something that's very specific to the workloads
> you are targetting.
>

Thanks for the feedback. I will find the performance delta first to decide
any such option makes sense.



> --
> Thanks,
> Anatoly
>