[dpdk-users] Multiple DPDK application instances on the same machine?

[dpdk-users] Multiple DPDK application instances on the same machine?

[Templin (US), Fred L]

Hi, we have a need to run multiple instances of applications that use the DPDK library on the
same machine. We are currently working with only two instances, but can't even get that to
work because the two seem to interfere with each other in terms of mempool creation. In
the future, we are going to want to have many more instances of the app all running in the
same physical/virtual machine. Is there a way to run multiple DPDK-using applications on
the same machine?

We note that there is only one /sys/kernel/mm/hugepages/ area per machine, and our
instructions were for *each application* to set the number of hugepages. Could it be that
there needs to be a way for the application instances to cooperatively share this area
instead of interfering with one another?

Thanks - Fred

Re: [dpdk-users] Multiple DPDK application instances on the same machine?

[Trahe, Fiona]

Hi Fred,

There may be other ways, but this is what I do:
I use --file-prefix <some_unique_name> and -m <some usubset of the total huge pages>
so I can run multiple instances of the tests in parallel.
e.g.
./dpdk/app/test --file-prefix myProcess1 -m 128  (plus some other options)
./dpdk/app/test --file-prefix myProcess2 -m 128  

The prefix is used to mark separate memory allocations and the -m limits the amount of memory used by the process so there's some available for other processes.
Hope this helps
Fiona



> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Templin (US), Fred L
> Sent: Wednesday, October 9, 2019 6:07 PM
> To: users <users@dpdk.org>
> Subject: [dpdk-users] Multiple DPDK application instances on the same machine?
> 
> Hi, we have a need to run multiple instances of applications that use the DPDK library on the
> same machine. We are currently working with only two instances, but can't even get that to
> work because the two seem to interfere with each other in terms of mempool creation. In
> the future, we are going to want to have many more instances of the app all running in the
> same physical/virtual machine. Is there a way to run multiple DPDK-using applications on
> the same machine?
> 
> We note that there is only one /sys/kernel/mm/hugepages/ area per machine, and our
> instructions were for *each application* to set the number of hugepages. Could it be that
> there needs to be a way for the application instances to cooperatively share this area
> instead of interfering with one another?
> 
> Thanks - Fred

Re: [dpdk-users] Multiple DPDK application instances on the same machine?

[Wiles, Keith]

> On Oct 9, 2019, at 12:49 PM, Trahe, Fiona <fiona.trahe@intel.com> wrote:
> 
> Hi Fred,
> 
> There may be other ways, but this is what I do:
> I use --file-prefix <some_unique_name> and -m <some usubset of the total huge pages>
> so I can run multiple instances of the tests in parallel.
> e.g.
> ./dpdk/app/test --file-prefix myProcess1 -m 128  (plus some other options)
> ./dpdk/app/test --file-prefix myProcess2 -m 128  
> 
> The prefix is used to mark separate memory allocations and the -m limits the amount of memory used by the process so there's some available for other processes.
> Hope this helps
> Fiona

Three resources need to be segmented and assigned to each DPDK instance, memory, logical core and devices.

- With memory the —file-prefix is defining a different set of huge page files by prefixing myProcess1 instead of the default rte_ to each huge page file.
The -m option limits the amount of memory the instance can use, which is the total amount of memory in huge pages divided by the number of instances (if each requires the same amount of memory).
- Logical cores must also be split up among the instances use the -l option
- If you are using resources like physical ports then you need to give each instance its own set of resources. You will need to --blacklist or --whitelist the physical devices for each instance.

Adding —blacklist (-b) or whitelist (-w) to the command line.

With ports at PCI address 83:00.0 and 83:00.1 and only those bound to DPDK (igb_uio, vfio, …) if you have more bound to DPDK and only using two than blacklist the extra devices.

Minimum amount of memory require for the two instances below is 256M

# use 83:00.0 with instance 1
./dpdk/app/test -l 1-4 —file-prefix foo1 -m 128 -b 83:00.1

# use 83:00.1 with instance 2
./dpdk/app/test -l 5-8 —file-prefix foo2 -m 128 -b 83:00.0

You can use whitelist also, just can't use both on the same command line.

HTH

> 
> 
> 
>> -----Original Message-----
>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Templin (US), Fred L
>> Sent: Wednesday, October 9, 2019 6:07 PM
>> To: users <users@dpdk.org>
>> Subject: [dpdk-users] Multiple DPDK application instances on the same machine?
>> 
>> Hi, we have a need to run multiple instances of applications that use the DPDK library on the
>> same machine. We are currently working with only two instances, but can't even get that to
>> work because the two seem to interfere with each other in terms of mempool creation. In
>> the future, we are going to want to have many more instances of the app all running in the
>> same physical/virtual machine. Is there a way to run multiple DPDK-using applications on
>> the same machine?
>> 
>> We note that there is only one /sys/kernel/mm/hugepages/ area per machine, and our
>> instructions were for *each application* to set the number of hugepages. Could it be that
>> there needs to be a way for the application instances to cooperatively share this area
>> instead of interfering with one another?
>> 
>> Thanks - Fred

Regards,
Keith

Re: [dpdk-users] Multiple DPDK application instances on the same machine?

[Templin (US), Fred L]

Many thanks, Keith and Fiona - we will test using your suggestions and see how we
progress from here, but it sounds promising.

Best regards - Fred

> -----Original Message-----
> From: Wiles, Keith [mailto:keith.wiles@intel.com]
> Sent: Wednesday, October 09, 2019 2:54 PM
> To: Trahe, Fiona <fiona.trahe@intel.com>
> Cc: Templin (US), Fred L <Fred.L.Templin@boeing.com>; users <users@dpdk.org>
> Subject: Re: [dpdk-users] Multiple DPDK application instances on the same machine?
> 
> 
> 
> > On Oct 9, 2019, at 12:49 PM, Trahe, Fiona <fiona.trahe@intel.com> wrote:
> >
> > Hi Fred,
> >
> > There may be other ways, but this is what I do:
> > I use --file-prefix <some_unique_name> and -m <some usubset of the total huge pages>
> > so I can run multiple instances of the tests in parallel.
> > e.g.
> > ./dpdk/app/test --file-prefix myProcess1 -m 128  (plus some other options)
> > ./dpdk/app/test --file-prefix myProcess2 -m 128
> >
> > The prefix is used to mark separate memory allocations and the -m limits the amount of memory used by the process so there's
> some available for other processes.
> > Hope this helps
> > Fiona
> 
> Three resources need to be segmented and assigned to each DPDK instance, memory, logical core and devices.
> 
> - With memory the —file-prefix is defining a different set of huge page files by prefixing myProcess1 instead of the default rte_ to
> each huge page file.
> The -m option limits the amount of memory the instance can use, which is the total amount of memory in huge pages divided by the
> number of instances (if each requires the same amount of memory).
> - Logical cores must also be split up among the instances use the -l option
> - If you are using resources like physical ports then you need to give each instance its own set of resources. You will need to --blacklist
> or --whitelist the physical devices for each instance.
> 
> Adding —blacklist (-b) or whitelist (-w) to the command line.
> 
> With ports at PCI address 83:00.0 and 83:00.1 and only those bound to DPDK (igb_uio, vfio, …) if you have more bound to DPDK and
> only using two than blacklist the extra devices.
> 
> Minimum amount of memory require for the two instances below is 256M
> 
> # use 83:00.0 with instance 1
> ./dpdk/app/test -l 1-4 —file-prefix foo1 -m 128 -b 83:00.1
> 
> # use 83:00.1 with instance 2
> ./dpdk/app/test -l 5-8 —file-prefix foo2 -m 128 -b 83:00.0
> 
> You can use whitelist also, just can't use both on the same command line.
> 
> HTH
> 
> >
> >
> >
> >> -----Original Message-----
> >> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Templin (US), Fred L
> >> Sent: Wednesday, October 9, 2019 6:07 PM
> >> To: users <users@dpdk.org>
> >> Subject: [dpdk-users] Multiple DPDK application instances on the same machine?
> >>
> >> Hi, we have a need to run multiple instances of applications that use the DPDK library on the
> >> same machine. We are currently working with only two instances, but can't even get that to
> >> work because the two seem to interfere with each other in terms of mempool creation. In
> >> the future, we are going to want to have many more instances of the app all running in the
> >> same physical/virtual machine. Is there a way to run multiple DPDK-using applications on
> >> the same machine?
> >>
> >> We note that there is only one /sys/kernel/mm/hugepages/ area per machine, and our
> >> instructions were for *each application* to set the number of hugepages. Could it be that
> >> there needs to be a way for the application instances to cooperatively share this area
> >> instead of interfering with one another?
> >>
> >> Thanks - Fred
> 
> Regards,
> Keith