[dpdk-dev] RTE_EAL on single core CPUs

[dpdk-dev] RTE_EAL on single core CPUs

[Marc Sune]

Dear all,

I was preparing a development machine (kvm - qemu) with a single core, 
and stumbled with what appears to be a limitation with EAL [1]. The VM 
is setup emulating a SandyBridge CPU but with a single CPU and running 
1.6.0 branch HEAD (perhaps this is the problem?¿).

I was also interested in this particular setup, because we haven't yet 
tried our application with some Atom equipment we have here, but we need 
to make it run also there.

Any ideas? I am probably missing something really fundamental here.

Thank you and regards
marc

[1]

marc@dpdk:~/dpdk/examples/l2fwd/build$ sudo ./l2fwd  -c 1 -n 1 -- -p=0xf
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Skip lcore 1 (not detected)
EAL: Skip lcore 2 (not detected)
EAL: Skip lcore 3 (not detected)
EAL: Skip lcore 4 (not detected)
EAL: Skip lcore 5 (not detected)
EAL: Skip lcore 6 (not detected)
EAL: Skip lcore 7 (not detected)
EAL: Skip lcore 8 (not detected)
EAL: Skip lcore 9 (not detected)
EAL: Skip lcore 10 (not detected)
EAL: Skip lcore 11 (not detected)
EAL: Skip lcore 12 (not detected)
EAL: Skip lcore 13 (not detected)
EAL: Skip lcore 14 (not detected)
EAL: Skip lcore 15 (not detected)
EAL: Skip lcore 16 (not detected)
EAL: Skip lcore 17 (not detected)
EAL: Skip lcore 18 (not detected)
EAL: Skip lcore 19 (not detected)
EAL: Skip lcore 20 (not detected)
EAL: Skip lcore 21 (not detected)
EAL: Skip lcore 22 (not detected)
EAL: Skip lcore 23 (not detected)
EAL: Skip lcore 24 (not detected)
EAL: Skip lcore 25 (not detected)
EAL: Skip lcore 26 (not detected)
EAL: Skip lcore 27 (not detected)
EAL: Skip lcore 28 (not detected)
EAL: Skip lcore 29 (not detected)
EAL: Skip lcore 30 (not detected)
EAL: Skip lcore 31 (not detected)
EAL: Skip lcore 32 (not detected)
EAL: Skip lcore 33 (not detected)
EAL: Skip lcore 34 (not detected)
EAL: Skip lcore 35 (not detected)
EAL: Skip lcore 36 (not detected)
EAL: Skip lcore 37 (not detected)
EAL: Skip lcore 38 (not detected)
EAL: Skip lcore 39 (not detected)
EAL: Skip lcore 40 (not detected)
EAL: Skip lcore 41 (not detected)
EAL: Skip lcore 42 (not detected)
EAL: Skip lcore 43 (not detected)
EAL: Skip lcore 44 (not detected)
EAL: Skip lcore 45 (not detected)
EAL: Skip lcore 46 (not detected)
EAL: Skip lcore 47 (not detected)
EAL: Skip lcore 48 (not detected)
EAL: Skip lcore 49 (not detected)
EAL: Skip lcore 50 (not detected)
EAL: Skip lcore 51 (not detected)
EAL: Skip lcore 52 (not detected)
EAL: Skip lcore 53 (not detected)
EAL: Skip lcore 54 (not detected)
EAL: Skip lcore 55 (not detected)
EAL: Skip lcore 56 (not detected)
EAL: Skip lcore 57 (not detected)
EAL: Skip lcore 58 (not detected)
EAL: Skip lcore 59 (not detected)
EAL: Skip lcore 60 (not detected)
EAL: Skip lcore 61 (not detected)
EAL: Skip lcore 62 (not detected)
EAL: Skip lcore 63 (not detected)
EAL: Setting up memory...
EAL: Ask a virtual area of 0x5c00000 bytes
EAL: Virtual area found at 0x7fbfaf600000 (size = 0x5c00000)
EAL: Ask a virtual area of 0x1800000 bytes
EAL: Virtual area found at 0x7fbfadc00000 (size = 0x1800000)
EAL: Ask a virtual area of 0x600000 bytes
EAL: Virtual area found at 0x7fbfad400000 (size = 0x600000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fbfad000000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fbfacc00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fbfac800000 (size = 0x200000)
EAL: Requesting 64 pages of size 2MB from socket 0
EAL: TSC frequency is ~2494382 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using 
unreliable clock cycles !
EAL: Master core 0 is ready (tid=b6129840)
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   PCI memory mapped at 0x7fbfb60d3000
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   PCI memory mapped at 0x7fbfb60b3000
EAL: PCI device 0000:00:08.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   PCI memory mapped at 0x7fbfb6051000
EAL: PCI device 0000:00:09.0 on NUMA socket -1
EAL:   probe driver: 8086:100e rte_em_pmd
EAL:   PCI memory mapped at 0x7fbfb6031000
Lcore 0: RX port 0
EAL: Error - exiting with code: 1
   Cause: Not enough cores
marc@dpdk:~/dpdk/examples/l2fwd/build$

Re: [dpdk-dev] RTE_EAL on single core CPUs

[Richardson, Bruce]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Marc Sune
> Sent: Monday, April 07, 2014 1:51 PM
> To: <dev@dpdk.org>
> Subject: [dpdk-dev] RTE_EAL on single core CPUs
>
> Dear all,
>
> I was preparing a development machine (kvm - qemu) with a single core, and stumbled with 
> what appears to be a limitation with EAL [1]. 
> The VM is setup emulating a SandyBridge CPU but with a single CPU and running
> 1.6.0 branch HEAD (perhaps this is the problem?¿).
>
> I was also interested in this particular setup, because we haven't yet tried our application 
> with some Atom equipment we have here, but we need to make it run also there.
>
> Any ideas? I am probably missing something really fundamental here.

Hi Marc,

I think in your case you've hit more a limitation of the particular app, rather than one for the EAL. L2fwd requires more than a single core to run, but you can easily write applications that can handle packets from multiple ports using a single core. 
Where you may hit issues, though, is that you cannot isolate the single core cpu from the linux kernel, so you may need to ensure you have enough buffering throughout the app to avoid packet loss when the kernel interrupts you to do its own house-keeping tasks.

Regards,
/Bruce

Re: [dpdk-dev] RTE_EAL on single core CPUs

[Marc Sune]

Dear Bruce,

Thank you for the quick reply, you are indeed right.

In fact the problem is much more simple than that, now that I realise. 
Our application must have one (or more) threads which do background 
tasks aside from packet processing (one of them is actually controlling 
the I/O), and with multi-core architectures we were sacrifising core0 
for such purpose.

The question now would be how is the appropriate way to treat this 
situation with DPDK;

a) First creating a thread before calling rte_eal_init():

main thread -> pthread_create() I/O thread -> call rte_eal_init() (from 
I/O thread? main thread? irrelevant?) -> ... -> I/O thread calls 
rte_eal_remote_launch() to launch itself

b) Create it after:

main thread -> rte_eal_init() call from main thread -> pthread_create() 
I/O thread  -> ... -> I/O thread calls rte_remote_launch() to launch itself

I guess option a) would be more suitable, is it?

thank you and regards
marc

On 07/04/14 14:55, Richardson, Bruce wrote:
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Marc Sune
>> Sent: Monday, April 07, 2014 1:51 PM
>> To: <dev@dpdk.org>
>> Subject: [dpdk-dev] RTE_EAL on single core CPUs
>>
>> Dear all,
>>
>> I was preparing a development machine (kvm - qemu) with a single core, and stumbled with
>> what appears to be a limitation with EAL [1].
>> The VM is setup emulating a SandyBridge CPU but with a single CPU and running
>> 1.6.0 branch HEAD (perhaps this is the problem?¿).
>>
>> I was also interested in this particular setup, because we haven't yet tried our application
>> with some Atom equipment we have here, but we need to make it run also there.
>>
>> Any ideas? I am probably missing something really fundamental here.
> Hi Marc,
>
> I think in your case you've hit more a limitation of the particular app, rather than one for the EAL. L2fwd requires more than a single core to run, but you can easily write applications that can handle packets from multiple ports using a single core.
> Where you may hit issues, though, is that you cannot isolate the single core cpu from the linux kernel, so you may need to ensure you have enough buffering throughout the app to avoid packet loss when the kernel interrupts you to do its own house-keeping tasks.
>
> Regards,
> /Bruce
>