[dpdk-dev] Issues running Ethtool on KNI interfaces

[dpdk-dev] Issues running Ethtool on KNI interfaces

[Deshvanth Mirle Jayaprakash (dmirleja)]

[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]

Hi,

Hypervisor: KVM

VM: Linux OS with 2.6.32 Kernel

VM Settings:

8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1

10Gigi Passthrough Interfaces attached to VM: Intel X520


DPDK settings:


DPDK Version: 1.6R2


echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

rmmod ixgbe

insmod igb_uio.ko

mkdir /mnt/hugepages

mount -t hugetlbfs nodev /mnt/hugepages

#./igb_uio_bind.py --status

#./igb_uio_bind.py --bind=igb_uio 03:01.0

#./igb_uio_bind.py --bind=igb_uio 03:02.0

#./igb_uio_bind.py --status

modprobe hwmon

insmod rte_kni.ko

./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" &


Having KNI process running in the background, creates two KNI interfaces called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached to the Userspace. The issue is that the Ethtool doesn't work with KNI interfaces. Is there anything that I am missing, How can I get Ethtool to work with KNI interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI application consumes all the huge pages(It doesn't mater 512/4096) assigned in the very first step.Is that expected behavior? Can I get KNI to work without assigning any Hugepages?



Also, I have attached screen shots of KNI log file. Please take a look.


Thanks,

Desh

[-- Attachment #2: KNI-SS4.png --]
[-- Type: image/png, Size: 12209 bytes --]

[-- Attachment #3: KNI-SS1.png --]
[-- Type: image/png, Size: 15281 bytes --]

[-- Attachment #4: KNI-SS2.png --]
[-- Type: image/png, Size: 15239 bytes --]

[-- Attachment #5: KNI-SS3.png --]
[-- Type: image/png, Size: 12812 bytes --]

Re: [dpdk-dev] Issues running Ethtool on KNI interfaces

[Zhang, Helin]

Hi Desh

You tried to use ethtool for KNI interfaces in VM, right? I don't think it is supported in VM.
Currently it just supports ethtool for KNI interfaces in host for some igb and ixgbe NICs.

Regards,
Helin

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle
> Jayaprakash (dmirleja)
> Sent: Monday, October 13, 2014 3:21 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces
> 
> Hi,
> 
> Hypervisor: KVM
> 
> VM: Linux OS with 2.6.32 Kernel
> 
> VM Settings:
> 
> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config
> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1
> 
> 10Gigi Passthrough Interfaces attached to VM: Intel X520
> 
> 
> DPDK settings:
> 
> 
> DPDK Version: 1.6R2
> 
> 
> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> 
> rmmod ixgbe
> 
> insmod igb_uio.ko
> 
> mkdir /mnt/hugepages
> 
> mount -t hugetlbfs nodev /mnt/hugepages
> 
> #./igb_uio_bind.py --status
> 
> #./igb_uio_bind.py --bind=igb_uio 03:01.0
> 
> #./igb_uio_bind.py --bind=igb_uio 03:02.0
> 
> #./igb_uio_bind.py --status
> 
> modprobe hwmon
> 
> insmod rte_kni.ko
> 
> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" &
> 
> 
> Having KNI process running in the background, creates two KNI interfaces
> called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached to
> the Userspace. The issue is that the Ethtool doesn't work with KNI interfaces.
> Is there anything that I am missing, How can I get Ethtool to work with KNI
> interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI application
> consumes all the huge pages(It doesn't mater 512/4096) assigned in the very
> first step.Is that expected behavior? Can I get KNI to work without assigning
> any Hugepages?
> 
> 
> 
> Also, I have attached screen shots of KNI log file. Please take a look.
> 
> 
> Thanks,
> 
> Desh

Re: [dpdk-dev] Issues running Ethtool on KNI interfaces

[Deshvanth Mirle Jayaprakash (dmirleja)]

Thanks Helin, have been trying this for some time, Is there any other way
I can pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1
? Can I use IOCTLS on these references in Kernel to gather interface
statistics, set MTU or bring up/down UIO interfaces.

/Desh

On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:

>Hi Desh
>
>You tried to use ethtool for KNI interfaces in VM, right? I don't think
>it is supported in VM.
>Currently it just supports ethtool for KNI interfaces in host for some
>igb and ixgbe NICs.
>
>Regards,
>Helin
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle
>> Jayaprakash (dmirleja)
>> Sent: Monday, October 13, 2014 3:21 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces
>> 
>> Hi,
>> 
>> Hypervisor: KVM
>> 
>> VM: Linux OS with 2.6.32 Kernel
>> 
>> VM Settings:
>> 
>> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config
>> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4, Threads=1
>> 
>> 10Gigi Passthrough Interfaces attached to VM: Intel X520
>> 
>> 
>> DPDK settings:
>> 
>> 
>> DPDK Version: 1.6R2
>> 
>> 
>> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>> 
>> rmmod ixgbe
>> 
>> insmod igb_uio.ko
>> 
>> mkdir /mnt/hugepages
>> 
>> mount -t hugetlbfs nodev /mnt/hugepages
>> 
>> #./igb_uio_bind.py --status
>> 
>> #./igb_uio_bind.py --bind=igb_uio 03:01.0
>> 
>> #./igb_uio_bind.py --bind=igb_uio 03:02.0
>> 
>> #./igb_uio_bind.py --status
>> 
>> modprobe hwmon
>> 
>> insmod rte_kni.ko
>> 
>> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3
>>--config="(0,1,3),(1,2,4)" &
>> 
>> 
>> Having KNI process running in the background, creates two KNI interfaces
>> called vEth0 and vEth1 which corresponds to IGB-UIO interfaces attached
>>to
>> the Userspace. The issue is that the Ethtool doesn't work with KNI
>>interfaces.
>> Is there anything that I am missing, How can I get Ethtool to work with
>>KNI
>> interfaces? Also, If I do not use --socket-mem OR -m EAL option, KNI
>>application
>> consumes all the huge pages(It doesn't mater 512/4096) assigned in the
>>very
>> first step.Is that expected behavior? Can I get KNI to work without
>>assigning
>> any Hugepages?
>> 
>> 
>> 
>> Also, I have attached screen shots of KNI log file. Please take a look.
>> 
>> 
>> Thanks,
>> 
>> Desh

Re: [dpdk-dev] Issues running Ethtool on KNI interfaces

[Zhang, Helin]

Hi Desh

Actually KNI provide a path to exchange info/actions between user space and kernel space. You can read kni example application and KNI kernel module and KNI library for more details. It already supports setting MTU, link up/down the port, etc.

Regards,
Helin

> -----Original Message-----
> From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com]
> Sent: Monday, October 13, 2014 3:39 PM
> To: Zhang, Helin; dev@dpdk.org
> Subject: Re: Issues running Ethtool on KNI interfaces
> 
> Thanks Helin, have been trying this for some time, Is there any other way I can
> pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1 ? Can I
> use IOCTLS on these references in Kernel to gather interface statistics, set
> MTU or bring up/down UIO interfaces.
> 
> /Desh
> 
> On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:
> 
> >Hi Desh
> >
> >You tried to use ethtool for KNI interfaces in VM, right? I don't think
> >it is supported in VM.
> >Currently it just supports ethtool for KNI interfaces in host for some
> >igb and ixgbe NICs.
> >
> >Regards,
> >Helin
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle
> >> Jayaprakash (dmirleja)
> >> Sent: Monday, October 13, 2014 3:21 PM
> >> To: dev@dpdk.org
> >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces
> >>
> >> Hi,
> >>
> >> Hypervisor: KVM
> >>
> >> VM: Linux OS with 2.6.32 Kernel
> >>
> >> VM Settings:
> >>
> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config
> >> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4,
> >> Threads=1
> >>
> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520
> >>
> >>
> >> DPDK settings:
> >>
> >>
> >> DPDK Version: 1.6R2
> >>
> >>
> >> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> >>
> >> rmmod ixgbe
> >>
> >> insmod igb_uio.ko
> >>
> >> mkdir /mnt/hugepages
> >>
> >> mount -t hugetlbfs nodev /mnt/hugepages
> >>
> >> #./igb_uio_bind.py --status
> >>
> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0
> >>
> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0
> >>
> >> #./igb_uio_bind.py --status
> >>
> >> modprobe hwmon
> >>
> >> insmod rte_kni.ko
> >>
> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3
> >>--config="(0,1,3),(1,2,4)" &
> >>
> >>
> >> Having KNI process running in the background, creates two KNI
> >>interfaces  called vEth0 and vEth1 which corresponds to IGB-UIO
> >>interfaces attached to  the Userspace. The issue is that the Ethtool
> >>doesn't work with KNI interfaces.
> >> Is there anything that I am missing, How can I get Ethtool to work
> >>with KNI  interfaces? Also, If I do not use --socket-mem OR -m EAL
> >>option, KNI application  consumes all the huge pages(It doesn't mater
> >>512/4096) assigned in the very  first step.Is that expected behavior?
> >>Can I get KNI to work without assigning  any Hugepages?
> >>
> >>
> >>
> >> Also, I have attached screen shots of KNI log file. Please take a look.
> >>
> >>
> >> Thanks,
> >>
> >> Desh

Re: [dpdk-dev] Issues running Ethtool on KNI interfaces

[Deshvanth Mirle Jayaprakash (dmirleja)]

Hi Helin,

  We have been using KNI application to create Virtual interfaces. We
would like to use KNI interfaces for control plane application only. As
intended, We are still planning to use UIO interfaces to pass Data plane
traffic which would bump up traffic throughput. We have assigned 4 vCPUs
for KNI application (CoreMask is 0x1E (Core 1,2,3, and 4)). When I run top
while KNI process is running in the background, see there is 100% CPU
utilization on all 4 Cores.

 Is there a way to fix this or is this expected? In our case, the CPU
utilization seems to cause lot of issues wrt to other processes running on
Linux OS. Am I missing any EAL option in KNI.Also, I was wondering If
there is a way to run KNI without Hugepages.

The Command:

./kni -c 0x1E -n 4 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" &

Thanks,
Desh

 There is some issue with respect to Hugepage allocation that prevents
other processes from
On 10/13/14 12:46 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:

>Hi Desh
>
>Actually KNI provide a path to exchange info/actions between user space
>and kernel space. You can read kni example application and KNI kernel
>module and KNI library for more details. It already supports setting MTU,
>link up/down the port, etc.
>
>Regards,
>Helin
>
>> -----Original Message-----
>> From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com]
>> Sent: Monday, October 13, 2014 3:39 PM
>> To: Zhang, Helin; dev@dpdk.org
>> Subject: Re: Issues running Ethtool on KNI interfaces
>> 
>> Thanks Helin, have been trying this for some time, Is there any other
>>way I can
>> pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and /dev/uio1 ?
>>Can I
>> use IOCTLS on these references in Kernel to gather interface
>>statistics, set
>> MTU or bring up/down UIO interfaces.
>> 
>> /Desh
>> 
>> On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:
>> 
>> >Hi Desh
>> >
>> >You tried to use ethtool for KNI interfaces in VM, right? I don't think
>> >it is supported in VM.
>> >Currently it just supports ethtool for KNI interfaces in host for some
>> >igb and ixgbe NICs.
>> >
>> >Regards,
>> >Helin
>> >
>> >> -----Original Message-----
>> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth Mirle
>> >> Jayaprakash (dmirleja)
>> >> Sent: Monday, October 13, 2014 3:21 PM
>> >> To: dev@dpdk.org
>> >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces
>> >>
>> >> Hi,
>> >>
>> >> Hypervisor: KVM
>> >>
>> >> VM: Linux OS with 2.6.32 Kernel
>> >>
>> >> VM Settings:
>> >>
>> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU Config
>> >> (SandyBridge), Manually set CPU topology: Sockets=2, Cores=4,
>> >> Threads=1
>> >>
>> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520
>> >>
>> >>
>> >> DPDK settings:
>> >>
>> >>
>> >> DPDK Version: 1.6R2
>> >>
>> >>
>> >> echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
>> >>
>> >> rmmod ixgbe
>> >>
>> >> insmod igb_uio.ko
>> >>
>> >> mkdir /mnt/hugepages
>> >>
>> >> mount -t hugetlbfs nodev /mnt/hugepages
>> >>
>> >> #./igb_uio_bind.py --status
>> >>
>> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0
>> >>
>> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0
>> >>
>> >> #./igb_uio_bind.py --status
>> >>
>> >> modprobe hwmon
>> >>
>> >> insmod rte_kni.ko
>> >>
>> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3
>> >>--config="(0,1,3),(1,2,4)" &
>> >>
>> >>
>> >> Having KNI process running in the background, creates two KNI
>> >>interfaces  called vEth0 and vEth1 which corresponds to IGB-UIO
>> >>interfaces attached to  the Userspace. The issue is that the Ethtool
>> >>doesn't work with KNI interfaces.
>> >> Is there anything that I am missing, How can I get Ethtool to work
>> >>with KNI  interfaces? Also, If I do not use --socket-mem OR -m EAL
>> >>option, KNI application  consumes all the huge pages(It doesn't mater
>> >>512/4096) assigned in the very  first step.Is that expected behavior?
>> >>Can I get KNI to work without assigning  any Hugepages?
>> >>
>> >>
>> >>
>> >> Also, I have attached screen shots of KNI log file. Please take a
>>look.
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> Desh
>

Re: [dpdk-dev] Issues running Ethtool on KNI interfaces

[Zhang, Helin]

Hi Desh

> -----Original Message-----
> From: Deshvanth Mirle Jayaprakash (dmirleja) [mailto:dmirleja@cisco.com]
> Sent: Friday, October 17, 2014 3:58 PM
> To: Zhang, Helin; dev@dpdk.org
> Subject: Re: Issues running Ethtool on KNI interfaces
> 
> Hi Helin,
> 
>   We have been using KNI application to create Virtual interfaces. We would
> like to use KNI interfaces for control plane application only. As intended, We are
> still planning to use UIO interfaces to pass Data plane traffic which would bump
> up traffic throughput. We have assigned 4 vCPUs for KNI application (CoreMask
> is 0x1E (Core 1,2,3, and 4)). When I run top while KNI process is running in the
> background, see there is 100% CPU utilization on all 4 Cores.
Yes, all user space threads are 100% utilized in DPDK, including KNI user space threads.
I guess somebody is thinking of a good solution for this issue, but for now, it is what we can expected.

> 
>  Is there a way to fix this or is this expected? In our case, the CPU utilization
> seems to cause lot of issues wrt to other processes running on Linux OS. Am I
> missing any EAL option in KNI.Also, I was wondering If there is a way to run KNI
> without Hugepages.
See my answer above, KNI user space app is actually a special DPDK application, it is based on all DPDK technologies, including hugepages.

> 
> The Command:
> 
> ./kni -c 0x1E -n 4 --socket-mem 512 -- -p 0x3 --config="(0,1,3),(1,2,4)" &
> 
> Thanks,
> Desh
> 
>  There is some issue with respect to Hugepage allocation that prevents other
> processes from On 10/13/14 12:46 AM, "Zhang, Helin"
> <helin.zhang@intel.com> wrote:
> 
> >Hi Desh
> >
> >Actually KNI provide a path to exchange info/actions between user space
> >and kernel space. You can read kni example application and KNI kernel
> >module and KNI library for more details. It already supports setting
> >MTU, link up/down the port, etc.
> >
> >Regards,
> >Helin
> >
> >> -----Original Message-----
> >> From: Deshvanth Mirle Jayaprakash (dmirleja)
> >> [mailto:dmirleja@cisco.com]
> >> Sent: Monday, October 13, 2014 3:39 PM
> >> To: Zhang, Helin; dev@dpdk.org
> >> Subject: Re: Issues running Ethtool on KNI interfaces
> >>
> >> Thanks Helin, have been trying this for some time, Is there any other
> >>way I can  pass IOCTLs to IGB-UIO interfaces. Can I use /dev/uio0 and
> >>/dev/uio1 ?
> >>Can I
> >> use IOCTLS on these references in Kernel to gather interface
> >>statistics, set  MTU or bring up/down UIO interfaces.
> >>
> >> /Desh
> >>
> >> On 10/13/14 12:34 AM, "Zhang, Helin" <helin.zhang@intel.com> wrote:
> >>
> >> >Hi Desh
> >> >
> >> >You tried to use ethtool for KNI interfaces in VM, right? I don't
> >> >think it is supported in VM.
> >> >Currently it just supports ethtool for KNI interfaces in host for
> >> >some igb and ixgbe NICs.
> >> >
> >> >Regards,
> >> >Helin
> >> >
> >> >> -----Original Message-----
> >> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Deshvanth
> >> >> Mirle Jayaprakash (dmirleja)
> >> >> Sent: Monday, October 13, 2014 3:21 PM
> >> >> To: dev@dpdk.org
> >> >> Subject: [dpdk-dev] Issues running Ethtool on KNI interfaces
> >> >>
> >> >> Hi,
> >> >>
> >> >> Hypervisor: KVM
> >> >>
> >> >> VM: Linux OS with 2.6.32 Kernel
> >> >>
> >> >> VM Settings:
> >> >>
> >> >> 8 vCPUs, 8192 MB of memory, CPU Configuration: Copy Host CPU
> >> >> Config (SandyBridge), Manually set CPU topology: Sockets=2,
> >> >> Cores=4,
> >> >> Threads=1
> >> >>
> >> >> 10Gigi Passthrough Interfaces attached to VM: Intel X520
> >> >>
> >> >>
> >> >> DPDK settings:
> >> >>
> >> >>
> >> >> DPDK Version: 1.6R2
> >> >>
> >> >>
> >> >> echo 512 >
> /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
> >> >>
> >> >> rmmod ixgbe
> >> >>
> >> >> insmod igb_uio.ko
> >> >>
> >> >> mkdir /mnt/hugepages
> >> >>
> >> >> mount -t hugetlbfs nodev /mnt/hugepages
> >> >>
> >> >> #./igb_uio_bind.py --status
> >> >>
> >> >> #./igb_uio_bind.py --bind=igb_uio 03:01.0
> >> >>
> >> >> #./igb_uio_bind.py --bind=igb_uio 03:02.0
> >> >>
> >> >> #./igb_uio_bind.py --status
> >> >>
> >> >> modprobe hwmon
> >> >>
> >> >> insmod rte_kni.ko
> >> >>
> >> >> ./kni -c 0x1E -n 2 --socket-mem 512 -- -p 0x3
> >> >>--config="(0,1,3),(1,2,4)" &
> >> >>
> >> >>
> >> >> Having KNI process running in the background, creates two KNI
> >> >>interfaces  called vEth0 and vEth1 which corresponds to IGB-UIO
> >> >>interfaces attached to  the Userspace. The issue is that the
> >> >>Ethtool doesn't work with KNI interfaces.
> >> >> Is there anything that I am missing, How can I get Ethtool to work
> >> >>with KNI  interfaces? Also, If I do not use --socket-mem OR -m EAL
> >> >>option, KNI application  consumes all the huge pages(It doesn't
> >> >>mater
> >> >>512/4096) assigned in the very  first step.Is that expected behavior?
> >> >>Can I get KNI to work without assigning  any Hugepages?
> >> >>
> >> >>
> >> >>
> >> >> Also, I have attached screen shots of KNI log file. Please take a
> >>look.
> >> >>
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Desh
> >

Regards,
Helin