[dpdk-dev] packet loss: multi-queue (RSS enabled)

[dpdk-dev] packet loss: multi-queue (RSS enabled)

[Hamid Ramazani]

Hi,
I tried a lot (more than a week) to solve the problem myself and not
to bother the list, but I didn't succeed. Maybe other people have the
same problem.

I have a simple program attached. It is intended for simple packet
capturing; captures from interface and writes to memory, and frees the
memory in the next loop.

I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing packets.
As you may know, this network card supports up to 128 RSS queues.

This is just a test, so the packets being sent at 820Kpps (kilo packet
per second). Each packet is 1500B (fixed size); it is 9.16 Gbit per
second.
Of course when the packet per seconds goes up and packet size goes
down (e.g. 400B per packet), it gets much worse.

When using one queue to receive, I receive all the packets, with no loss.
When I use more than one queue (e.g. 8 queues), with each thread
running on a dedicated core, I have a considerable amount of loss.

Please note that:
1. The computer has 12 * 2.67GHz cores, and it does nothing else but
capturing packets. The CPU is Intel Xeon X5650.
2. The operating system is Ubuntu 12.04.3 LTS

Attached file includes:
main.h
main.c
Makefile
./run.sh

It is configured to be run with 8 queues.

If you want to change the number of receive queues, please:
1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired value.
2. change core mask in run.sh file (since there is SKIP_MASTER, you
should give a core containing one more CPU than given number of
queues).

I think there might be following problems:
1. the port configuration is not fine.
2. freeing memory has a considerable amount of overhead, and may be I
shouldn't do that. But If I don't the pool will be full, won't be? Is
there any other way?

Please help.
Thanks a lot in advance for your help and comments.

All the Best,
Hamid

Re: [dpdk-dev] packet loss: multi-queue (RSS enabled)

[Jayakumar, Muthurajan]

Hi, 
Please find the attached paper http://kfall.net/ucbpage/papers/snc.pdf
Figures 4 and 5 refers about the degradation when the # of queues are increased. 
It refers sweet spot as 2 to 4 queues. 

Have you please verified with smaller # of queues please?

Thanks, 


-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hamid Ramazani
Sent: Tuesday, April 29, 2014 9:42 PM
To: Thomas Monjalon; dev@dpdk.org
Subject: [dpdk-dev] packet loss: multi-queue (RSS enabled)

Hi,
I tried a lot (more than a week) to solve the problem myself and not to bother the list, but I didn't succeed. Maybe other people have the same problem.

I have a simple program attached. It is intended for simple packet capturing; captures from interface and writes to memory, and frees the memory in the next loop.

I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing packets.
As you may know, this network card supports up to 128 RSS queues.

This is just a test, so the packets being sent at 820Kpps (kilo packet per second). Each packet is 1500B (fixed size); it is 9.16 Gbit per second.
Of course when the packet per seconds goes up and packet size goes down (e.g. 400B per packet), it gets much worse.

When using one queue to receive, I receive all the packets, with no loss.
When I use more than one queue (e.g. 8 queues), with each thread running on a dedicated core, I have a considerable amount of loss.

Please note that:
1. The computer has 12 * 2.67GHz cores, and it does nothing else but capturing packets. The CPU is Intel Xeon X5650.
2. The operating system is Ubuntu 12.04.3 LTS

Attached file includes:
main.h
main.c
Makefile
./run.sh

It is configured to be run with 8 queues.

If you want to change the number of receive queues, please:
1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired value.
2. change core mask in run.sh file (since there is SKIP_MASTER, you should give a core containing one more CPU than given number of queues).

I think there might be following problems:
1. the port configuration is not fine.
2. freeing memory has a considerable amount of overhead, and may be I shouldn't do that. But If I don't the pool will be full, won't be? Is there any other way?

Please help.
Thanks a lot in advance for your help and comments.

All the Best,
Hamid

Re: [dpdk-dev] packet loss: multi-queue (RSS enabled)

[Daniel Kaminsky]

Hi Hamid,

I didn't see any attachment but I think there is a solution. My first
question is, when you created the mempool, how much did you defined the
memory cache per CPU? In the example program this number is fairly small
(32) and I think that increasing it to something much bigger (e.g. 512)
will significantly improve the CPU scalability.

Regards,
Daniel Kaminsky


On Wed, Apr 30, 2014 at 7:56 AM, Jayakumar, Muthurajan <
muthurajan.jayakumar@intel.com> wrote:

> Hi,
> Please find the attached paper http://kfall.net/ucbpage/papers/snc.pdf
> Figures 4 and 5 refers about the degradation when the # of queues are
> increased.
> It refers sweet spot as 2 to 4 queues.
>
> Have you please verified with smaller # of queues please?
>
> Thanks,
>
>
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Hamid Ramazani
> Sent: Tuesday, April 29, 2014 9:42 PM
> To: Thomas Monjalon; dev@dpdk.org
> Subject: [dpdk-dev] packet loss: multi-queue (RSS enabled)
>
> Hi,
> I tried a lot (more than a week) to solve the problem myself and not to
> bother the list, but I didn't succeed. Maybe other people have the same
> problem.
>
> I have a simple program attached. It is intended for simple packet
> capturing; captures from interface and writes to memory, and frees the
> memory in the next loop.
>
> I have a 10G 82599EB Intel SFI/SFP+ network interface for capturing
> packets.
> As you may know, this network card supports up to 128 RSS queues.
>
> This is just a test, so the packets being sent at 820Kpps (kilo packet per
> second). Each packet is 1500B (fixed size); it is 9.16 Gbit per second.
> Of course when the packet per seconds goes up and packet size goes down
> (e.g. 400B per packet), it gets much worse.
>
> When using one queue to receive, I receive all the packets, with no loss.
> When I use more than one queue (e.g. 8 queues), with each thread running
> on a dedicated core, I have a considerable amount of loss.
>
> Please note that:
> 1. The computer has 12 * 2.67GHz cores, and it does nothing else but
> capturing packets. The CPU is Intel Xeon X5650.
> 2. The operating system is Ubuntu 12.04.3 LTS
>
> Attached file includes:
> main.h
> main.c
> Makefile
> ./run.sh
>
> It is configured to be run with 8 queues.
>
> If you want to change the number of receive queues, please:
> 1. in main.c, change the value assigned to nb_rx_q_of_dev to the desired
> value.
> 2. change core mask in run.sh file (since there is SKIP_MASTER, you should
> give a core containing one more CPU than given number of queues).
>
> I think there might be following problems:
> 1. the port configuration is not fine.
> 2. freeing memory has a considerable amount of overhead, and may be I
> shouldn't do that. But If I don't the pool will be full, won't be? Is there
> any other way?
>
> Please help.
> Thanks a lot in advance for your help and comments.
>
> All the Best,
> Hamid
>