[dpdk-dev] Packet Rx issue with DPDK1.8

[dpdk-dev] Packet Rx issue with DPDK1.8

[Prashant Upadhyaya]

Hi,

I am migrating from DPDK1.7 to DPDK1.8.
My application works fine with DPDK1.7.
I am using 10 Gb Intel 82599 NIC.
I have jumbo frames enabled, with max_rx_pkt_len = 10232
My mbuf dataroom size is 2048+headroom
So naturally the ixgbe_recv_scattered_pkts driver function is triggered for
receiving.
This works with DPDK1.7 and my app receives packets.
However, it does not work with DPDK1.8 somehow.I don't receive any packets.

So, I increased the mbuf data room size in my application to a higher value
so that the function ixgbe_recv_scattered_pkts is not enabled (I believe
ixgbe_recv_pkts will be used in this case), and now my application starts
getting packets with DPDK1.8 and the entire application usecase works fine
(ofcourse my application had to adapt to the mbuf structure changes which I
have done)

I am kind of coming to the conclusion that ixgbe_recv_scattered_pkts has
something broken in DPDK1.8 as compared to the earlier versions by the
above empirical evidence.

Has anybody else faced a similar issue ?

Regards
-Prashant

Re: [dpdk-dev] Packet Rx issue with DPDK1.8

[Bruce Richardson]

On Thu, Jan 08, 2015 at 01:40:54PM +0530, Prashant Upadhyaya wrote:
> Hi,
> 
> I am migrating from DPDK1.7 to DPDK1.8.
> My application works fine with DPDK1.7.
> I am using 10 Gb Intel 82599 NIC.
> I have jumbo frames enabled, with max_rx_pkt_len = 10232
> My mbuf dataroom size is 2048+headroom
> So naturally the ixgbe_recv_scattered_pkts driver function is triggered for
> receiving.
> This works with DPDK1.7 and my app receives packets.
> However, it does not work with DPDK1.8 somehow.I don't receive any packets.
> 
> So, I increased the mbuf data room size in my application to a higher value
> so that the function ixgbe_recv_scattered_pkts is not enabled (I believe
> ixgbe_recv_pkts will be used in this case), and now my application starts
> getting packets with DPDK1.8 and the entire application usecase works fine
> (ofcourse my application had to adapt to the mbuf structure changes which I
> have done)
> 
> I am kind of coming to the conclusion that ixgbe_recv_scattered_pkts has
> something broken in DPDK1.8 as compared to the earlier versions by the
> above empirical evidence.
> 
> Has anybody else faced a similar issue ?
> 
> Regards
> -Prashant

This is worrying to hear. In 1.8, there is now the receive_scattered_pkts_vec
function which manages changed mbufs. This was tested - both in development and
in validation - before release, but since it's new code, it's entirely possible
we missed something. Can you perhaps try disabling the vector driver in 1.8,
and see if receiving scattered packets/chained mbufs works? 

Regards,
/Bruce

Re: [dpdk-dev] Packet Rx issue with DPDK1.8

[Prashant Upadhyaya]

Hi Bruce,

I tried with your suggestion.

When I disable the _vec function with the following config change, the usecase works for me. So it points to some issue in the _vec function.

CONFIG_RTE_IXGBE_INC_VECTOR=y, I changed this parameter to CONFIG_RTE_IXGBE_INC_VECTOR=n

There appears to be some gottcha in the function therefore, somebody may want to run some tests again perhaps with jumbo frames enabled (and sending small normal frames)

Regards
-Prashant

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
Sent: Thursday, January 08, 2015 7:17 PM
To: Prashant Upadhyaya
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] Packet Rx issue with DPDK1.8

On Thu, Jan 08, 2015 at 01:40:54PM +0530, Prashant Upadhyaya wrote:
> Hi,
>
> I am migrating from DPDK1.7 to DPDK1.8.
> My application works fine with DPDK1.7.
> I am using 10 Gb Intel 82599 NIC.
> I have jumbo frames enabled, with max_rx_pkt_len = 10232 My mbuf
> dataroom size is 2048+headroom So naturally the
> ixgbe_recv_scattered_pkts driver function is triggered for receiving.
> This works with DPDK1.7 and my app receives packets.
> However, it does not work with DPDK1.8 somehow.I don't receive any packets.
>
> So, I increased the mbuf data room size in my application to a higher
> value so that the function ixgbe_recv_scattered_pkts is not enabled (I
> believe ixgbe_recv_pkts will be used in this case), and now my
> application starts getting packets with DPDK1.8 and the entire
> application usecase works fine (ofcourse my application had to adapt
> to the mbuf structure changes which I have done)
>
> I am kind of coming to the conclusion that ixgbe_recv_scattered_pkts
> has something broken in DPDK1.8 as compared to the earlier versions by
> the above empirical evidence.
>
> Has anybody else faced a similar issue ?
>
> Regards
> -Prashant

This is worrying to hear. In 1.8, there is now the receive_scattered_pkts_vec function which manages changed mbufs. This was tested - both in development and in validation - before release, but since it's new code, it's entirely possible we missed something. Can you perhaps try disabling the vector driver in 1.8, and see if receiving scattered packets/chained mbufs works?

Regards,
/Bruce
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Re: [dpdk-dev] Packet Rx issue with DPDK1.8

[Bruce Richardson]

On Fri, Jan 09, 2015 at 06:45:57AM +0000, Prashant Upadhyaya wrote:
> Hi Bruce,
> 
> I tried with your suggestion.
> 
> When I disable the _vec function with the following config change, the usecase works for me. So it points to some issue in the _vec function.
> 
> CONFIG_RTE_IXGBE_INC_VECTOR=y, I changed this parameter to CONFIG_RTE_IXGBE_INC_VECTOR=n
> 
> There appears to be some gottcha in the function therefore, somebody may want to run some tests again perhaps with jumbo frames enabled (and sending small normal frames)
> 
> Regards
> -Prashant
> 

Yes, we'll perform additional testing and see what we can turn up.

/Bruce

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Bruce Richardson
> Sent: Thursday, January 08, 2015 7:17 PM
> To: Prashant Upadhyaya
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Packet Rx issue with DPDK1.8
> 
> On Thu, Jan 08, 2015 at 01:40:54PM +0530, Prashant Upadhyaya wrote:
> > Hi,
> >
> > I am migrating from DPDK1.7 to DPDK1.8.
> > My application works fine with DPDK1.7.
> > I am using 10 Gb Intel 82599 NIC.
> > I have jumbo frames enabled, with max_rx_pkt_len = 10232 My mbuf
> > dataroom size is 2048+headroom So naturally the
> > ixgbe_recv_scattered_pkts driver function is triggered for receiving.
> > This works with DPDK1.7 and my app receives packets.
> > However, it does not work with DPDK1.8 somehow.I don't receive any packets.
> >
> > So, I increased the mbuf data room size in my application to a higher
> > value so that the function ixgbe_recv_scattered_pkts is not enabled (I
> > believe ixgbe_recv_pkts will be used in this case), and now my
> > application starts getting packets with DPDK1.8 and the entire
> > application usecase works fine (ofcourse my application had to adapt
> > to the mbuf structure changes which I have done)
> >
> > I am kind of coming to the conclusion that ixgbe_recv_scattered_pkts
> > has something broken in DPDK1.8 as compared to the earlier versions by
> > the above empirical evidence.
> >
> > Has anybody else faced a similar issue ?
> >
> > Regards
> > -Prashant
> 
> This is worrying to hear. In 1.8, there is now the receive_scattered_pkts_vec function which manages changed mbufs. This was tested - both in development and in validation - before release, but since it's new code, it's entirely possible we missed something. Can you perhaps try disabling the vector driver in 1.8, and see if receiving scattered packets/chained mbufs works?
> 
> Regards,
> /Bruce
> "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

Re: [dpdk-dev] Packet Rx issue with DPDK1.8

[Thomas Monjalon]

2015-01-09 09:26, Bruce Richardson:
> On Fri, Jan 09, 2015 at 06:45:57AM +0000, Prashant Upadhyaya wrote:
> > Hi Bruce,
> > 
> > I tried with your suggestion.
> > 
> > When I disable the _vec function with the following config change, the
> > usecase works for me. So it points to some issue in the _vec function.
> > 
> > CONFIG_RTE_IXGBE_INC_VECTOR=y, I changed this parameter to
> > CONFIG_RTE_IXGBE_INC_VECTOR=n
> > 
> > There appears to be some gottcha in the function therefore, somebody may
> > want to run some tests again perhaps with jumbo frames enabled (and
> > sending small normal frames)
> > 
> > Regards
> > -Prashant
> 
> Yes, we'll perform additional testing and see what we can turn up.

Bruce, any news about this bug?
It seems important and could justify (with other bugs) to release a version
1.8.1.

Thanks
-- 
Thomas

Re: [dpdk-dev] Packet Rx issue with DPDK1.8

[Bruce Richardson]

On Wed, Jan 14, 2015 at 05:30:15PM +0100, Thomas Monjalon wrote:
> 2015-01-09 09:26, Bruce Richardson:
> > On Fri, Jan 09, 2015 at 06:45:57AM +0000, Prashant Upadhyaya wrote:
> > > Hi Bruce,
> > > 
> > > I tried with your suggestion.
> > > 
> > > When I disable the _vec function with the following config change, the
> > > usecase works for me. So it points to some issue in the _vec function.
> > > 
> > > CONFIG_RTE_IXGBE_INC_VECTOR=y, I changed this parameter to
> > > CONFIG_RTE_IXGBE_INC_VECTOR=n
> > > 
> > > There appears to be some gottcha in the function therefore, somebody may
> > > want to run some tests again perhaps with jumbo frames enabled (and
> > > sending small normal frames)
> > > 
> > > Regards
> > > -Prashant
> > 
> > Yes, we'll perform additional testing and see what we can turn up.
> 
> Bruce, any news about this bug?
> It seems important and could justify (with other bugs) to release a version
> 1.8.1.
> 
> Thanks
> -- 
> Thomas

No updates on my end yet. I'll keep you informed as I get any extra info here.

/Bruce

[dpdk-dev] New to DPDK

[Ravi Rao]

Hi All,
    I am a newbee to DPDK. Can one of you please let me know if there is 
any reference board that is available which I can use to build and 
tryout the dpdk stuff on.
Regards,
Ravi

Re: [dpdk-dev] New to DPDK

[Wiles, Keith]

Most people I guess use a Xeon CPU class MB with one or two sockets
running Linux with a supported NICs. I use a motherboard like the one
below running Ubuntu 12.04 with 12G RAM and the 82599 NICs. You can find
more supported NICs in the documentation and you need find the rest of the
parts :-) You do not need much disk space I have a 500G disk and you can
use less memory, but that is something you need to decide on.

http://www.intel.com/content/www/us/en/motherboards/server-motherboards/ser
ver-board-w2600cr.html


On 1/14/15, 12:33 PM, "Ravi Rao" <Ravi.Rao@us.fujitsu.com> wrote:

>Hi All,
>    I am a newbee to DPDK. Can one of you please let me know if there is
>any reference board that is available which I can use to build and
>tryout the dpdk stuff on.
>Regards,
>Ravi

Re: [dpdk-dev] New to DPDK

[Ravi Rao]

Thanks a lot for the quick response.
On 01/14/2015 01:27 PM, Wiles, Keith wrote:
> Most people I guess use a Xeon CPU class MB with one or two sockets
> running Linux with a supported NICs. I use a motherboard like the one
> below running Ubuntu 12.04 with 12G RAM and the 82599 NICs. You can find
> more supported NICs in the documentation and you need find the rest of the
> parts :-) You do not need much disk space I have a 500G disk and you can
> use less memory, but that is something you need to decide on.
>
> http://www.intel.com/content/www/us/en/motherboards/server-motherboards/ser
> ver-board-w2600cr.html
>
>
> On 1/14/15, 12:33 PM, "Ravi Rao" <Ravi.Rao@us.fujitsu.com> wrote:
>
>> Hi All,
>>     I am a newbee to DPDK. Can one of you please let me know if there is
>> any reference board that is available which I can use to build and
>> tryout the dpdk stuff on.
>> Regards,
>> Ravi

Re: [dpdk-dev] New to DPDK

[Ravi Rao]

Hi,
    Thanks for the inputs. I did download dpdk-1.8 and used your latest 
packet-gen-2.8.0 but I am getting this crash.
Looks like I am missing or not specifying the correct arguments. Can you 
please help ?
./app/pktgen -c 03 -n 1 --proc-type auto --socket-mem 128 --file-prefix 
pg -- -T -P -m "1.0, 2.1" -f themes/black-yellow.theme

!PANIC!: Cannot configure device: port=0, Num queues 2,2 (2)Invalid argument
PANIC in pktgen_config_ports():
Cannot configure device: port=0, Num queues 2,2 (2)Invalid argument6: 
[./app/pktgen() [0x4268c3]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) 
[0x7f6959e0dec5]]
4: [./app/pktgen(main+0x47c) [0x4262ac]]
3: [./app/pktgen(pktgen_config_ports+0x16ab) [0x43e09b]]
2: [./app/pktgen(__rte_panic+0xc1) [0x425d20]]
1: [./app/pktgen(rte_dump_stack+0x18) [0x4bce48]]
./doit.sh: line 53: 21912 Aborted                 (core dumped) 
./app/pktgen -c 0f -n 3 --proc-type auto --socket-mem 128 --file-prefix 
pg -- -T -P -m "1.0, 2.1" -f themes/black-yellow.theme
labadmin@Openstack:~/dpdk-1.8.0/pktgen-2.8.0$ sudo gdb ./app/pktgen 
-core core
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./app/pktgen...done.
[New LWP 21912]
[New LWP 21924]
[New LWP 21925]
[New LWP 21923]
[New LWP 21922]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./app/pktgen -c 0f -n 3 --proc-type auto 
--socket-mem 128 --file-prefix pg -- -'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f6959e22cc9 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007f6959e22cc9 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f6959e260d8 in __GI_abort () at abort.c:89
#2  0x0000000000425d25 in __rte_panic ()
#3  0x000000000043e09b in pktgen_config_ports () at pktgen-port-cfg.c:275
#4  0x00000000004262ac in main (argc=7, argv=0x7fff12510090) at 
pktgen-main.c:420
(gdb) info shared
>From                To                  Syms Read   Shared Object Library
0x00007f695aae6c20  0x00007f695ab050fa  Yes (*) 
/usr/lib/x86_64-linux-gnu/libpcap.so.0.8
0x00007f695a7e1610  0x00007f695a8501b6  Yes /lib/x86_64-linux-gnu/libm.so.6
0x00007f695a5d6350  0x00007f695a5d933c  Yes /lib/x86_64-linux-gnu/librt.so.1
0x00007f695a3d0ed0  0x00007f695a3d19ce  Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007f695a1b79f0  0x00007f695a1c44a1  Yes 
/lib/x86_64-linux-gnu/libpthread.so.0
0x00007f6959e0b4a0  0x00007f6959f51113  Yes /lib/x86_64-linux-gnu/libc.so.6
0x00007f695ad20ae0  0x00007f695ad3b4e0  Yes /lib64/ld-linux-x86-64.so.2
0x00007f6959bd8ab0  0x00007f6959be8995  Yes (*) 
/lib/x86_64-linux-gnu/libgcc_s.so.1
(*): Shared library is missing debugging information.
(gdb)

Regards,
Ravi
On 01/14/2015 02:26 PM, Wiles, Keith wrote:
> BTW, you may want to check out the Pktgen-DPDK (I wrote) to give you a
> simple starting point or at least a traffic generator like system.
>
> http://dpdk.org/browse/apps/pktgen-dpdk/refs/
>
>
> On 1/14/15, 1:54 PM, "Ravi Rao" <Ravi.Rao@us.fujitsu.com> wrote:
>
>> Thanks a lot for the quick response.
>> On 01/14/2015 01:27 PM, Wiles, Keith wrote:
>>> Most people I guess use a Xeon CPU class MB with one or two sockets
>>> running Linux with a supported NICs. I use a motherboard like the one
>>> below running Ubuntu 12.04 with 12G RAM and the 82599 NICs. You can find
>>> more supported NICs in the documentation and you need find the rest of
>>> the
>>> parts :-) You do not need much disk space I have a 500G disk and you can
>>> use less memory, but that is something you need to decide on.
>>>
>>>
>>> http://www.intel.com/content/www/us/en/motherboards/server-motherboards/s
>>> er
>>> ver-board-w2600cr.html
>>>
>>>
>>> On 1/14/15, 12:33 PM, "Ravi Rao" <Ravi.Rao@us.fujitsu.com> wrote:
>>>
>>>> Hi All,
>>>>      I am a newbee to DPDK. Can one of you please let me know if there
>>>> is
>>>> any reference board that is available which I can use to build and
>>>> tryout the dpdk stuff on.
>>>> Regards,
>>>> Ravi