[dpdk-users] Run testpmd application and encountered no free page issues

[dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Hi,

I have one board with two 10G etherent on it. ( and two 1G ethernet as well
)

Below is my working environment:

OS: Ubuntu 16.04
10G Ethernet controller: X552/X557-AT 10GBASE-T
Ethernet driver: ixgbe - 4.2.1-k
dpdk version: 17.02.1

In order to avoid doing configuration incorrectly, i run the dpdk-setup.sh
in the directory usertools.

Press enter to continue ...
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armv8a-linuxapp-gcc
[2] arm64-dpaa2-linuxapp-gcc
[3] arm64-thunderx-linuxapp-gcc
[4] arm64-xgene1-linuxapp-gcc
[5] arm-armv7a-linuxapp-gcc
[6] i686-native-linuxapp-gcc
[7] i686-native-linuxapp-icc
[8] ppc_64-power8-linuxapp-gcc
[9] tile-tilegx-linuxapp-gcc
[10] x86_64-native-bsdapp-clang
[11] x86_64-native-bsdapp-gcc
[12] x86_64-native-linuxapp-clang
[13] x86_64-native-linuxapp-gcc
[14] x86_64-native-linuxapp-icc
[15] x86_x32-native-linuxapp-gcc

----------------------------------------------------------
 Step 2: Setup linuxapp environment
----------------------------------------------------------
[16] Insert IGB UIO module
[17] Insert VFIO module
[18] Insert KNI module
[19] Setup hugepage mappings for non-NUMA systems
[20] Setup hugepage mappings for NUMA systems
[21] Display current Ethernet/Crypto device settings
[22] Bind Ethernet/Crypto device to IGB UIO module
[23] Bind Ethernet/Crypto device to VFIO module
[24] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linuxapp environment
----------------------------------------------------------
[25] Run test application ($RTE_TARGET/app/test)
[26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[27] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[28] Unbind devices from IGB UIO or VFIO driver
[29] Remove IGB UIO module
[30] Remove VFIO module
[31] Remove KNI module
[32] Remove hugepage mappings

[33] Exit Script

*Option: 19*

Removing currently reserved hugepages
Unmounting /mnt/huge and removing directory

  Input the number of 2048kB hugepages
  Example: to have 128MB of hugepages available in a 2MB huge page system,
  enter '64' to reserve 64 * 2MB pages
Number of pages: 64
Reserving hugepages
Creating /mnt/huge and mounting as hugetlbfs

Press enter to continue ...
----------------------------------------------------------
 Step 1: Select the DPDK environment to build
----------------------------------------------------------
[1] arm64-armv8a-linuxapp-gcc
[2] arm64-dpaa2-linuxapp-gcc
[3] arm64-thunderx-linuxapp-gcc
[4] arm64-xgene1-linuxapp-gcc
[5] arm-armv7a-linuxapp-gcc
[6] i686-native-linuxapp-gcc
[7] i686-native-linuxapp-icc
[8] ppc_64-power8-linuxapp-gcc
[9] tile-tilegx-linuxapp-gcc
[10] x86_64-native-bsdapp-clang
[11] x86_64-native-bsdapp-gcc
[12] x86_64-native-linuxapp-clang
[13] x86_64-native-linuxapp-gcc
[14] x86_64-native-linuxapp-icc
[15] x86_x32-native-linuxapp-gcc

----------------------------------------------------------
 Step 2: Setup linuxapp environment
----------------------------------------------------------
[16] Insert IGB UIO module
[17] Insert VFIO module
[18] Insert KNI module
[19] Setup hugepage mappings for non-NUMA systems
[20] Setup hugepage mappings for NUMA systems
[21] Display current Ethernet/Crypto device settings
[22] Bind Ethernet/Crypto device to IGB UIO module
[23] Bind Ethernet/Crypto device to VFIO module
[24] Setup VFIO permissions

----------------------------------------------------------
 Step 3: Run test application for linuxapp environment
----------------------------------------------------------
[25] Run test application ($RTE_TARGET/app/test)
[26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)

----------------------------------------------------------
 Step 4: Other tools
----------------------------------------------------------
[27] List hugepage info from /proc/meminfo

----------------------------------------------------------
 Step 5: Uninstall and system cleanup
----------------------------------------------------------
[28] Unbind devices from IGB UIO or VFIO driver
[29] Remove IGB UIO module
[30] Remove VFIO module
[31] Remove KNI module
[32] Remove hugepage mappings

[33] Exit Script

*Option: 26*


  Enter hex bitmask of cores to execute testpmd app on
  Example: to execute app on cores 0 to 7, enter 0xff
bitmask: 0xff
Launching app
EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: No probed ethernet devices
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176,
socket=0
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory


Is there anything I might not do right ?

Your reply is appreciated.



Paul

-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Anyone,

or my question is not clear enough ?



Thanks




Paul

2017-06-16 18:20 GMT+08:00 Paul Tsvika <mozloverinweb@gmail.com>:

> Hi,
>
> I have one board with two 10G etherent on it. ( and two 1G ethernet as
> well )
>
> Below is my working environment:
>
> OS: Ubuntu 16.04
> 10G Ethernet controller: X552/X557-AT 10GBASE-T
> Ethernet driver: ixgbe - 4.2.1-k
> dpdk version: 17.02.1
>
> In order to avoid doing configuration incorrectly, i run the dpdk-setup.sh
> in the directory usertools.
>
> Press enter to continue ...
> ----------------------------------------------------------
>  Step 1: Select the DPDK environment to build
> ----------------------------------------------------------
> [1] arm64-armv8a-linuxapp-gcc
> [2] arm64-dpaa2-linuxapp-gcc
> [3] arm64-thunderx-linuxapp-gcc
> [4] arm64-xgene1-linuxapp-gcc
> [5] arm-armv7a-linuxapp-gcc
> [6] i686-native-linuxapp-gcc
> [7] i686-native-linuxapp-icc
> [8] ppc_64-power8-linuxapp-gcc
> [9] tile-tilegx-linuxapp-gcc
> [10] x86_64-native-bsdapp-clang
> [11] x86_64-native-bsdapp-gcc
> [12] x86_64-native-linuxapp-clang
> [13] x86_64-native-linuxapp-gcc
> [14] x86_64-native-linuxapp-icc
> [15] x86_x32-native-linuxapp-gcc
>
> ----------------------------------------------------------
>  Step 2: Setup linuxapp environment
> ----------------------------------------------------------
> [16] Insert IGB UIO module
> [17] Insert VFIO module
> [18] Insert KNI module
> [19] Setup hugepage mappings for non-NUMA systems
> [20] Setup hugepage mappings for NUMA systems
> [21] Display current Ethernet/Crypto device settings
> [22] Bind Ethernet/Crypto device to IGB UIO module
> [23] Bind Ethernet/Crypto device to VFIO module
> [24] Setup VFIO permissions
>
> ----------------------------------------------------------
>  Step 3: Run test application for linuxapp environment
> ----------------------------------------------------------
> [25] Run test application ($RTE_TARGET/app/test)
> [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
>
> ----------------------------------------------------------
>  Step 4: Other tools
> ----------------------------------------------------------
> [27] List hugepage info from /proc/meminfo
>
> ----------------------------------------------------------
>  Step 5: Uninstall and system cleanup
> ----------------------------------------------------------
> [28] Unbind devices from IGB UIO or VFIO driver
> [29] Remove IGB UIO module
> [30] Remove VFIO module
> [31] Remove KNI module
> [32] Remove hugepage mappings
>
> [33] Exit Script
>
> *Option: 19*
>
> Removing currently reserved hugepages
> Unmounting /mnt/huge and removing directory
>
>   Input the number of 2048kB hugepages
>   Example: to have 128MB of hugepages available in a 2MB huge page system,
>   enter '64' to reserve 64 * 2MB pages
> Number of pages: 64
> Reserving hugepages
> Creating /mnt/huge and mounting as hugetlbfs
>
> Press enter to continue ...
> ----------------------------------------------------------
>  Step 1: Select the DPDK environment to build
> ----------------------------------------------------------
> [1] arm64-armv8a-linuxapp-gcc
> [2] arm64-dpaa2-linuxapp-gcc
> [3] arm64-thunderx-linuxapp-gcc
> [4] arm64-xgene1-linuxapp-gcc
> [5] arm-armv7a-linuxapp-gcc
> [6] i686-native-linuxapp-gcc
> [7] i686-native-linuxapp-icc
> [8] ppc_64-power8-linuxapp-gcc
> [9] tile-tilegx-linuxapp-gcc
> [10] x86_64-native-bsdapp-clang
> [11] x86_64-native-bsdapp-gcc
> [12] x86_64-native-linuxapp-clang
> [13] x86_64-native-linuxapp-gcc
> [14] x86_64-native-linuxapp-icc
> [15] x86_x32-native-linuxapp-gcc
>
> ----------------------------------------------------------
>  Step 2: Setup linuxapp environment
> ----------------------------------------------------------
> [16] Insert IGB UIO module
> [17] Insert VFIO module
> [18] Insert KNI module
> [19] Setup hugepage mappings for non-NUMA systems
> [20] Setup hugepage mappings for NUMA systems
> [21] Display current Ethernet/Crypto device settings
> [22] Bind Ethernet/Crypto device to IGB UIO module
> [23] Bind Ethernet/Crypto device to VFIO module
> [24] Setup VFIO permissions
>
> ----------------------------------------------------------
>  Step 3: Run test application for linuxapp environment
> ----------------------------------------------------------
> [25] Run test application ($RTE_TARGET/app/test)
> [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
>
> ----------------------------------------------------------
>  Step 4: Other tools
> ----------------------------------------------------------
> [27] List hugepage info from /proc/meminfo
>
> ----------------------------------------------------------
>  Step 5: Uninstall and system cleanup
> ----------------------------------------------------------
> [28] Unbind devices from IGB UIO or VFIO driver
> [29] Remove IGB UIO module
> [30] Remove VFIO module
> [31] Remove KNI module
> [32] Remove hugepage mappings
>
> [33] Exit Script
>
> *Option: 26*
>
>
>   Enter hex bitmask of cores to execute testpmd app on
>   Example: to execute app on cores 0 to 7, enter 0xff
> bitmask: 0xff
> Launching app
> EAL: Detected 16 lcore(s)
> EAL: No free hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:03:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:05:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:05:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: No probed ethernet devices
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176,
> socket=0
> EAL: Error - exiting with code: 1
>   Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
>
>
> Is there anything I might not do right ?
>
> Your reply is appreciated.
>
>
>
> Paul
>
> --
> P.T
>



-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Shreyansh Jain]

Hello Paul,

Some comments inline:

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Paul Tsvika
> Sent: Saturday, June 17, 2017 9:17 PM
> To: users@dpdk.org
> Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> page issues
> 
> Anyone,
> 
> or my question is not clear enough ?
> 
> 
> 
> Thanks
> 
> 
> 
> 
> Paul
> 
> 2017-06-16 18:20 GMT+08:00 Paul Tsvika <mozloverinweb@gmail.com>:
> 
> > Hi,
> >
> > I have one board with two 10G etherent on it. ( and two 1G ethernet as
> > well )
> >
> > Below is my working environment:
> >
> > OS: Ubuntu 16.04
> > 10G Ethernet controller: X552/X557-AT 10GBASE-T
> > Ethernet driver: ixgbe - 4.2.1-k
> > dpdk version: 17.02.1
> >
> > In order to avoid doing configuration incorrectly, i run the dpdk-setup.sh
> > in the directory usertools.
> >
> > Press enter to continue ...
> > ----------------------------------------------------------
> >  Step 1: Select the DPDK environment to build
> > ----------------------------------------------------------
> > [1] arm64-armv8a-linuxapp-gcc
> > [2] arm64-dpaa2-linuxapp-gcc
> > [3] arm64-thunderx-linuxapp-gcc
> > [4] arm64-xgene1-linuxapp-gcc
> > [5] arm-armv7a-linuxapp-gcc
> > [6] i686-native-linuxapp-gcc
> > [7] i686-native-linuxapp-icc
> > [8] ppc_64-power8-linuxapp-gcc
> > [9] tile-tilegx-linuxapp-gcc
> > [10] x86_64-native-bsdapp-clang
> > [11] x86_64-native-bsdapp-gcc
> > [12] x86_64-native-linuxapp-clang
> > [13] x86_64-native-linuxapp-gcc
> > [14] x86_64-native-linuxapp-icc
> > [15] x86_x32-native-linuxapp-gcc
> >
> > ----------------------------------------------------------
> >  Step 2: Setup linuxapp environment
> > ----------------------------------------------------------
> > [16] Insert IGB UIO module
> > [17] Insert VFIO module
> > [18] Insert KNI module
> > [19] Setup hugepage mappings for non-NUMA systems
> > [20] Setup hugepage mappings for NUMA systems
> > [21] Display current Ethernet/Crypto device settings
> > [22] Bind Ethernet/Crypto device to IGB UIO module
> > [23] Bind Ethernet/Crypto device to VFIO module
> > [24] Setup VFIO permissions
> >
> > ----------------------------------------------------------
> >  Step 3: Run test application for linuxapp environment
> > ----------------------------------------------------------
> > [25] Run test application ($RTE_TARGET/app/test)
> > [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
> >
> > ----------------------------------------------------------
> >  Step 4: Other tools
> > ----------------------------------------------------------
> > [27] List hugepage info from /proc/meminfo
> >
> > ----------------------------------------------------------
> >  Step 5: Uninstall and system cleanup
> > ----------------------------------------------------------
> > [28] Unbind devices from IGB UIO or VFIO driver
> > [29] Remove IGB UIO module
> > [30] Remove VFIO module
> > [31] Remove KNI module
> > [32] Remove hugepage mappings
> >
> > [33] Exit Script
> >
> > *Option: 19*
> >
> > Removing currently reserved hugepages
> > Unmounting /mnt/huge and removing directory
> >
> >   Input the number of 2048kB hugepages
> >   Example: to have 128MB of hugepages available in a 2MB huge page system,
> >   enter '64' to reserve 64 * 2MB pages
> > Number of pages: 64
> > Reserving hugepages
> > Creating /mnt/huge and mounting as hugetlbfs
 
Essentially you are trying to reserve 64x2M = 128M memory. Did you try increasing it?

> >
> > Press enter to continue ...
> > ----------------------------------------------------------
> >  Step 1: Select the DPDK environment to build
> > ----------------------------------------------------------
> > [1] arm64-armv8a-linuxapp-gcc
> > [2] arm64-dpaa2-linuxapp-gcc
> > [3] arm64-thunderx-linuxapp-gcc
> > [4] arm64-xgene1-linuxapp-gcc
> > [5] arm-armv7a-linuxapp-gcc
> > [6] i686-native-linuxapp-gcc
> > [7] i686-native-linuxapp-icc
> > [8] ppc_64-power8-linuxapp-gcc
> > [9] tile-tilegx-linuxapp-gcc
> > [10] x86_64-native-bsdapp-clang
> > [11] x86_64-native-bsdapp-gcc
> > [12] x86_64-native-linuxapp-clang
> > [13] x86_64-native-linuxapp-gcc
> > [14] x86_64-native-linuxapp-icc
> > [15] x86_x32-native-linuxapp-gcc
> >
> > ----------------------------------------------------------
> >  Step 2: Setup linuxapp environment
> > ----------------------------------------------------------
> > [16] Insert IGB UIO module
> > [17] Insert VFIO module
> > [18] Insert KNI module
> > [19] Setup hugepage mappings for non-NUMA systems
> > [20] Setup hugepage mappings for NUMA systems
> > [21] Display current Ethernet/Crypto device settings
> > [22] Bind Ethernet/Crypto device to IGB UIO module
> > [23] Bind Ethernet/Crypto device to VFIO module
> > [24] Setup VFIO permissions
> >
> > ----------------------------------------------------------
> >  Step 3: Run test application for linuxapp environment
> > ----------------------------------------------------------
> > [25] Run test application ($RTE_TARGET/app/test)
> > [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
> >
> > ----------------------------------------------------------
> >  Step 4: Other tools
> > ----------------------------------------------------------
> > [27] List hugepage info from /proc/meminfo
> >
> > ----------------------------------------------------------
> >  Step 5: Uninstall and system cleanup
> > ----------------------------------------------------------
> > [28] Unbind devices from IGB UIO or VFIO driver
> > [29] Remove IGB UIO module
> > [30] Remove VFIO module
> > [31] Remove KNI module
> > [32] Remove hugepage mappings
> >
> > [33] Exit Script
> >
> > *Option: 26*
> >
> >
> >   Enter hex bitmask of cores to execute testpmd app on
> >   Example: to execute app on cores 0 to 7, enter 0xff
> > bitmask: 0xff
> > Launching app
> > EAL: Detected 16 lcore(s)
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:15ad net_ixgbe
> > EAL: PCI device 0000:03:00.1 on NUMA socket 0
> > EAL:   probe driver: 8086:15ad net_ixgbe
> > EAL: PCI device 0000:05:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 net_e1000_igb
> > EAL: PCI device 0000:05:00.1 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 net_e1000_igb
> > EAL: No probed ethernet devices
> > Interactive-mode selected
> > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176,
> > socket=0
 
Testpmd is trying to allocate 203456*2176 bytes ~ 442MB.
Can you try with memory more than this number?
Or, try with 'total-num-mbufs=<>' passed to testpmd.

> > EAL: Error - exiting with code: 1
> >   Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
> >
> >
> > Is there anything I might not do right ?
> >
> > Your reply is appreciated.
> >
> >
> >
> > Paul
> >
> > --
> > P.T
> >
> 
> 
> 
> --
> P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Hi Shreyansh,


Thanks for the reply.

I tried to increase the number from 64 to 230. And the number of 460 is
larger than 442.

However, the error message is the same.

Is there anything I can check here ?


Thanks



Paul

2017-06-19 13:49 GMT+08:00 Shreyansh Jain <shreyansh.jain@nxp.com>:

> Hello Paul,
>
> Some comments inline:
>
> > -----Original Message-----
> > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Paul Tsvika
> > Sent: Saturday, June 17, 2017 9:17 PM
> > To: users@dpdk.org
> > Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> > page issues
> >
> > Anyone,
> >
> > or my question is not clear enough ?
> >
> >
> >
> > Thanks
> >
> >
> >
> >
> > Paul
> >
> > 2017-06-16 18:20 GMT+08:00 Paul Tsvika <mozloverinweb@gmail.com>:
> >
> > > Hi,
> > >
> > > I have one board with two 10G etherent on it. ( and two 1G ethernet as
> > > well )
> > >
> > > Below is my working environment:
> > >
> > > OS: Ubuntu 16.04
> > > 10G Ethernet controller: X552/X557-AT 10GBASE-T
> > > Ethernet driver: ixgbe - 4.2.1-k
> > > dpdk version: 17.02.1
> > >
> > > In order to avoid doing configuration incorrectly, i run the
> dpdk-setup.sh
> > > in the directory usertools.
> > >
> > > Press enter to continue ...
> > > ----------------------------------------------------------
> > >  Step 1: Select the DPDK environment to build
> > > ----------------------------------------------------------
> > > [1] arm64-armv8a-linuxapp-gcc
> > > [2] arm64-dpaa2-linuxapp-gcc
> > > [3] arm64-thunderx-linuxapp-gcc
> > > [4] arm64-xgene1-linuxapp-gcc
> > > [5] arm-armv7a-linuxapp-gcc
> > > [6] i686-native-linuxapp-gcc
> > > [7] i686-native-linuxapp-icc
> > > [8] ppc_64-power8-linuxapp-gcc
> > > [9] tile-tilegx-linuxapp-gcc
> > > [10] x86_64-native-bsdapp-clang
> > > [11] x86_64-native-bsdapp-gcc
> > > [12] x86_64-native-linuxapp-clang
> > > [13] x86_64-native-linuxapp-gcc
> > > [14] x86_64-native-linuxapp-icc
> > > [15] x86_x32-native-linuxapp-gcc
> > >
> > > ----------------------------------------------------------
> > >  Step 2: Setup linuxapp environment
> > > ----------------------------------------------------------
> > > [16] Insert IGB UIO module
> > > [17] Insert VFIO module
> > > [18] Insert KNI module
> > > [19] Setup hugepage mappings for non-NUMA systems
> > > [20] Setup hugepage mappings for NUMA systems
> > > [21] Display current Ethernet/Crypto device settings
> > > [22] Bind Ethernet/Crypto device to IGB UIO module
> > > [23] Bind Ethernet/Crypto device to VFIO module
> > > [24] Setup VFIO permissions
> > >
> > > ----------------------------------------------------------
> > >  Step 3: Run test application for linuxapp environment
> > > ----------------------------------------------------------
> > > [25] Run test application ($RTE_TARGET/app/test)
> > > [26] Run testpmd application in interactive mode
> ($RTE_TARGET/app/testpmd)
> > >
> > > ----------------------------------------------------------
> > >  Step 4: Other tools
> > > ----------------------------------------------------------
> > > [27] List hugepage info from /proc/meminfo
> > >
> > > ----------------------------------------------------------
> > >  Step 5: Uninstall and system cleanup
> > > ----------------------------------------------------------
> > > [28] Unbind devices from IGB UIO or VFIO driver
> > > [29] Remove IGB UIO module
> > > [30] Remove VFIO module
> > > [31] Remove KNI module
> > > [32] Remove hugepage mappings
> > >
> > > [33] Exit Script
> > >
> > > *Option: 19*
> > >
> > > Removing currently reserved hugepages
> > > Unmounting /mnt/huge and removing directory
> > >
> > >   Input the number of 2048kB hugepages
> > >   Example: to have 128MB of hugepages available in a 2MB huge page
> system,
> > >   enter '64' to reserve 64 * 2MB pages
> > > Number of pages: 64
> > > Reserving hugepages
> > > Creating /mnt/huge and mounting as hugetlbfs
>
> Essentially you are trying to reserve 64x2M = 128M memory. Did you try
> increasing it?
>
> > >
> > > Press enter to continue ...
> > > ----------------------------------------------------------
> > >  Step 1: Select the DPDK environment to build
> > > ----------------------------------------------------------
> > > [1] arm64-armv8a-linuxapp-gcc
> > > [2] arm64-dpaa2-linuxapp-gcc
> > > [3] arm64-thunderx-linuxapp-gcc
> > > [4] arm64-xgene1-linuxapp-gcc
> > > [5] arm-armv7a-linuxapp-gcc
> > > [6] i686-native-linuxapp-gcc
> > > [7] i686-native-linuxapp-icc
> > > [8] ppc_64-power8-linuxapp-gcc
> > > [9] tile-tilegx-linuxapp-gcc
> > > [10] x86_64-native-bsdapp-clang
> > > [11] x86_64-native-bsdapp-gcc
> > > [12] x86_64-native-linuxapp-clang
> > > [13] x86_64-native-linuxapp-gcc
> > > [14] x86_64-native-linuxapp-icc
> > > [15] x86_x32-native-linuxapp-gcc
> > >
> > > ----------------------------------------------------------
> > >  Step 2: Setup linuxapp environment
> > > ----------------------------------------------------------
> > > [16] Insert IGB UIO module
> > > [17] Insert VFIO module
> > > [18] Insert KNI module
> > > [19] Setup hugepage mappings for non-NUMA systems
> > > [20] Setup hugepage mappings for NUMA systems
> > > [21] Display current Ethernet/Crypto device settings
> > > [22] Bind Ethernet/Crypto device to IGB UIO module
> > > [23] Bind Ethernet/Crypto device to VFIO module
> > > [24] Setup VFIO permissions
> > >
> > > ----------------------------------------------------------
> > >  Step 3: Run test application for linuxapp environment
> > > ----------------------------------------------------------
> > > [25] Run test application ($RTE_TARGET/app/test)
> > > [26] Run testpmd application in interactive mode
> ($RTE_TARGET/app/testpmd)
> > >
> > > ----------------------------------------------------------
> > >  Step 4: Other tools
> > > ----------------------------------------------------------
> > > [27] List hugepage info from /proc/meminfo
> > >
> > > ----------------------------------------------------------
> > >  Step 5: Uninstall and system cleanup
> > > ----------------------------------------------------------
> > > [28] Unbind devices from IGB UIO or VFIO driver
> > > [29] Remove IGB UIO module
> > > [30] Remove VFIO module
> > > [31] Remove KNI module
> > > [32] Remove hugepage mappings
> > >
> > > [33] Exit Script
> > >
> > > *Option: 26*
> > >
> > >
> > >   Enter hex bitmask of cores to execute testpmd app on
> > >   Example: to execute app on cores 0 to 7, enter 0xff
> > > bitmask: 0xff
> > > Launching app
> > > EAL: Detected 16 lcore(s)
> > > EAL: No free hugepages reported in hugepages-1048576kB
> > > EAL: Probing VFIO support...
> > > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > > EAL:   probe driver: 8086:15ad net_ixgbe
> > > EAL: PCI device 0000:03:00.1 on NUMA socket 0
> > > EAL:   probe driver: 8086:15ad net_ixgbe
> > > EAL: PCI device 0000:05:00.0 on NUMA socket 0
> > > EAL:   probe driver: 8086:1521 net_e1000_igb
> > > EAL: PCI device 0000:05:00.1 on NUMA socket 0
> > > EAL:   probe driver: 8086:1521 net_e1000_igb
> > > EAL: No probed ethernet devices
> > > Interactive-mode selected
> > > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456,
> size=2176,
> > > socket=0
>
> Testpmd is trying to allocate 203456*2176 bytes ~ 442MB.
> Can you try with memory more than this number?
> Or, try with 'total-num-mbufs=<>' passed to testpmd.
>
> > > EAL: Error - exiting with code: 1
> > >   Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate
> memory
> > >
> > >
> > > Is there anything I might not do right ?
> > >
> > > Your reply is appreciated.
> > >
> > >
> > >
> > > Paul
> > >
> > > --
> > > P.T
> > >
> >
> >
> >
> > --
> > P.T
>



-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Shreyansh Jain]

Hello Paul,

The error happens when testpmd application tries to acquire enough memory from the hugepages.
Only suggestion is to provide a little more memory (for example, 1g to start with) and see if that changes your situation.

How much memory can you spare for hugepages? Or, how much have you already spared?
Can you dump output of 'cat /proc/meminfo' (assuming Linux)?

I don't have any other suggestions beyond this.

-
Shreyansh

From: Paul Tsvika [mailto:mozloverinweb@gmail.com] 
Sent: Monday, June 19, 2017 11:38 AM
To: Shreyansh Jain <shreyansh.jain@nxp.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Run testpmd application and encountered no free page issues

Hi Shreyansh,


Thanks for the reply.

I tried to increase the number from 64 to 230. And the number of 460 is larger than 442.

However, the error message is the same.

Is there anything I can check here ?


Thanks



Paul

2017-06-19 13:49 GMT+08:00 Shreyansh Jain <mailto:shreyansh.jain@nxp.com>:
Hello Paul,

Some comments inline:

> -----Original Message-----
> From: users [mailto:mailto:users-bounces@dpdk.org] On Behalf Of Paul Tsvika
> Sent: Saturday, June 17, 2017 9:17 PM
> To: mailto:users@dpdk.org
> Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> page issues
>
> Anyone,
>
> or my question is not clear enough ?
>
>
>
> Thanks
>
>
>
>
> Paul
>
> 2017-06-16 18:20 GMT+08:00 Paul Tsvika <mailto:mozloverinweb@gmail.com>:
>
> > Hi,
> >
> > I have one board with two 10G etherent on it. ( and two 1G ethernet as
> > well )
> >
> > Below is my working environment:
> >
> > OS: Ubuntu 16.04
> > 10G Ethernet controller: X552/X557-AT 10GBASE-T
> > Ethernet driver: ixgbe - 4.2.1-k
> > dpdk version: 17.02.1
> >
> > In order to avoid doing configuration incorrectly, i run the dpdk-setup.sh
> > in the directory usertools.
> >
> > Press enter to continue ...
> > ----------------------------------------------------------
> >  Step 1: Select the DPDK environment to build
> > ----------------------------------------------------------
> > [1] arm64-armv8a-linuxapp-gcc
> > [2] arm64-dpaa2-linuxapp-gcc
> > [3] arm64-thunderx-linuxapp-gcc
> > [4] arm64-xgene1-linuxapp-gcc
> > [5] arm-armv7a-linuxapp-gcc
> > [6] i686-native-linuxapp-gcc
> > [7] i686-native-linuxapp-icc
> > [8] ppc_64-power8-linuxapp-gcc
> > [9] tile-tilegx-linuxapp-gcc
> > [10] x86_64-native-bsdapp-clang
> > [11] x86_64-native-bsdapp-gcc
> > [12] x86_64-native-linuxapp-clang
> > [13] x86_64-native-linuxapp-gcc
> > [14] x86_64-native-linuxapp-icc
> > [15] x86_x32-native-linuxapp-gcc
> >
> > ----------------------------------------------------------
> >  Step 2: Setup linuxapp environment
> > ----------------------------------------------------------
> > [16] Insert IGB UIO module
> > [17] Insert VFIO module
> > [18] Insert KNI module
> > [19] Setup hugepage mappings for non-NUMA systems
> > [20] Setup hugepage mappings for NUMA systems
> > [21] Display current Ethernet/Crypto device settings
> > [22] Bind Ethernet/Crypto device to IGB UIO module
> > [23] Bind Ethernet/Crypto device to VFIO module
> > [24] Setup VFIO permissions
> >
> > ----------------------------------------------------------
> >  Step 3: Run test application for linuxapp environment
> > ----------------------------------------------------------
> > [25] Run test application ($RTE_TARGET/app/test)
> > [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
> >
> > ----------------------------------------------------------
> >  Step 4: Other tools
> > ----------------------------------------------------------
> > [27] List hugepage info from /proc/meminfo
> >
> > ----------------------------------------------------------
> >  Step 5: Uninstall and system cleanup
> > ----------------------------------------------------------
> > [28] Unbind devices from IGB UIO or VFIO driver
> > [29] Remove IGB UIO module
> > [30] Remove VFIO module
> > [31] Remove KNI module
> > [32] Remove hugepage mappings
> >
> > [33] Exit Script
> >
> > *Option: 19*
> >
> > Removing currently reserved hugepages
> > Unmounting /mnt/huge and removing directory
> >
> >   Input the number of 2048kB hugepages
> >   Example: to have 128MB of hugepages available in a 2MB huge page system,
> >   enter '64' to reserve 64 * 2MB pages
> > Number of pages: 64
> > Reserving hugepages
> > Creating /mnt/huge and mounting as hugetlbfs

Essentially you are trying to reserve 64x2M = 128M memory. Did you try increasing it?

> >
> > Press enter to continue ...
> > ----------------------------------------------------------
> >  Step 1: Select the DPDK environment to build
> > ----------------------------------------------------------
> > [1] arm64-armv8a-linuxapp-gcc
> > [2] arm64-dpaa2-linuxapp-gcc
> > [3] arm64-thunderx-linuxapp-gcc
> > [4] arm64-xgene1-linuxapp-gcc
> > [5] arm-armv7a-linuxapp-gcc
> > [6] i686-native-linuxapp-gcc
> > [7] i686-native-linuxapp-icc
> > [8] ppc_64-power8-linuxapp-gcc
> > [9] tile-tilegx-linuxapp-gcc
> > [10] x86_64-native-bsdapp-clang
> > [11] x86_64-native-bsdapp-gcc
> > [12] x86_64-native-linuxapp-clang
> > [13] x86_64-native-linuxapp-gcc
> > [14] x86_64-native-linuxapp-icc
> > [15] x86_x32-native-linuxapp-gcc
> >
> > ----------------------------------------------------------
> >  Step 2: Setup linuxapp environment
> > ----------------------------------------------------------
> > [16] Insert IGB UIO module
> > [17] Insert VFIO module
> > [18] Insert KNI module
> > [19] Setup hugepage mappings for non-NUMA systems
> > [20] Setup hugepage mappings for NUMA systems
> > [21] Display current Ethernet/Crypto device settings
> > [22] Bind Ethernet/Crypto device to IGB UIO module
> > [23] Bind Ethernet/Crypto device to VFIO module
> > [24] Setup VFIO permissions
> >
> > ----------------------------------------------------------
> >  Step 3: Run test application for linuxapp environment
> > ----------------------------------------------------------
> > [25] Run test application ($RTE_TARGET/app/test)
> > [26] Run testpmd application in interactive mode ($RTE_TARGET/app/testpmd)
> >
> > ----------------------------------------------------------
> >  Step 4: Other tools
> > ----------------------------------------------------------
> > [27] List hugepage info from /proc/meminfo
> >
> > ----------------------------------------------------------
> >  Step 5: Uninstall and system cleanup
> > ----------------------------------------------------------
> > [28] Unbind devices from IGB UIO or VFIO driver
> > [29] Remove IGB UIO module
> > [30] Remove VFIO module
> > [31] Remove KNI module
> > [32] Remove hugepage mappings
> >
> > [33] Exit Script
> >
> > *Option: 26*
> >
> >
> >   Enter hex bitmask of cores to execute testpmd app on
> >   Example: to execute app on cores 0 to 7, enter 0xff
> > bitmask: 0xff
> > Launching app
> > EAL: Detected 16 lcore(s)
> > EAL: No free hugepages reported in hugepages-1048576kB
> > EAL: Probing VFIO support...
> > EAL: PCI device 0000:03:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:15ad net_ixgbe
> > EAL: PCI device 0000:03:00.1 on NUMA socket 0
> > EAL:   probe driver: 8086:15ad net_ixgbe
> > EAL: PCI device 0000:05:00.0 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 net_e1000_igb
> > EAL: PCI device 0000:05:00.1 on NUMA socket 0
> > EAL:   probe driver: 8086:1521 net_e1000_igb
> > EAL: No probed ethernet devices
> > Interactive-mode selected
> > USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176,
> > socket=0

Testpmd is trying to allocate 203456*2176 bytes ~ 442MB.
Can you try with memory more than this number?
Or, try with 'total-num-mbufs=<>' passed to testpmd.

> > EAL: Error - exiting with code: 1
> >   Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
> >
> >
> > Is there anything I might not do right ?
> >
> > Your reply is appreciated.
> >
> >
> >
> > Paul
> >
> > --
> > P.T
> >
>
>
>
> --
> P.T




-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Hi Shreyansh,

Thanks for the reply.

By using the set-up script in DPDK:

the hugepage memory looks to be allocated properly:

cat /proc/meminfo

HugePages_Total:     240
HugePages_Free:     240
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB


and then i run the commands below in sequence:

$ sudo modprobe uio
$ sudo insmod ./build/kmod/igb_uio.ko
$ sudo ./usertools/dpdk-devbind.py -b igb_uio xxx:xx.0 xxx:xx.1   (
xxx.xx.0, xxx.xx.1 <-- 10G port pci address )
$ sudo ./build/app/testpmd -l 1,2,3 -n 2 -- -i

EAL: Detected 16 lcore(s)
*EAL: No free hugepages reported in hugepages-1048576kB*
EAL: Probing VFIO support...
EAL: VFIO support initialized
*EAL: PCI device 0000:03:00.0 on NUMA socket 0*
*EAL:   probe driver: 8086:15ad net_ixgbe*
*EAL: PCI device 0000:03:00.1 on NUMA socket 0*
*EAL:   probe driver: 8086:15ad net_ixgbe*
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
socket=0
Configuring Port 0 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:25:90:5C:E9:58
Configuring Port 1 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:25:90:5C:E9:59
Checking link statuses...
Done
testpmd>


Questions below:

1. It looks like the application can enter the interactive mode. however, I
have no idea why *EAL: No free hugepages reported in hugepages-1048576kB *keeps
popping out.

2. Is there any reason why the ports went down why I run the commands? The
Link was up before running it.

3. The hugepage will be gone after running this application.

HugePages_Total:     240
HugePages_Free:         0
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB

It became this. However, how can i free it again ? I tried to umount and
mount again but it did not work.


Please advice if any.



Paul

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Shreyansh Jain]

Hello Paul,

Some comments inline, prepended with [SJ01]:
(Would it be possible for you to send text mails to this mailing list - it helps in keeping replies contextual)

From: Paul Tsvika [mailto:mozloverinweb@gmail.com] 
Sent: Tuesday, June 20, 2017 2:27 PM
To: Shreyansh Jain <shreyansh.jain@nxp.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Run testpmd application and encountered no free page issues

Hi Shreyansh,

Thanks for the reply.

By using the set-up script in DPDK:

the hugepage memory looks to be allocated properly:

cat /proc/meminfo

HugePages_Total:     240
HugePages_Free:     240
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB


and then i run the commands below in sequence:

$ sudo modprobe uio
$ sudo insmod ./build/kmod/igb_uio.ko
$ sudo ./usertools/dpdk-devbind.py -b igb_uio xxx:xx.0 xxx:xx.1   ( xxx.xx.0, xxx.xx.1 <-- 10G port pci address )
$ sudo ./build/app/testpmd -l 1,2,3 -n 2 -- -i

EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB

[SJ01] DPDK iterates over all directories in /sys/kernel/mm/hugepages which contains one directory for each supported hugepage size. This log above is appearing because nothing was found for a directory named "hugepages-1048576kB". That is probably because you have '0' hugepages of size 1G in your system. This should not impact you until you are were expecting 1G hugepage size.

EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
Configuring Port 0 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:25:90:5C:E9:58
Configuring Port 1 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:25:90:5C:E9:59
Checking link statuses...
Done
testpmd> 


Questions below:

1. It looks like the application can enter the interactive mode. however, I have no idea why EAL: No free hugepages reported in hugepages-1048576kB keeps popping out.

[SJ01] Lets ignore that for while. You have about 240 pages of 2M = ~480Mb. Application is demanding ~339Mb (n=163456, size=2176)


2. Is there any reason why the ports went down why I run the commands? The Link was up before running it.

[SJ01] This I am not sure. If the links were up _before_, they should be up now as well. Just out of curiosity, how did you check that links were up before? These links are not assigned to Linux kernel and are not visible in the ifconfig list.

3. The hugepage will be gone after running this application.

HugePages_Total:     240
HugePages_Free:         0
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB

It became this. However, how can i free it again ? I tried to umount and mount again but it did not work.

[SJ01] Just go and delete all files created in /mnt/hugepages/ folder. You would have "HugePages_Free" available again. But, only if you have the application stopped.


Please advice if any.

[SJ01] Above is all what I know. I have no idea why your links are appearing down.

-
Shreyansh

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Thanks Shreyansh.

My comment prepended with [P01]


2017-06-20 19:10 GMT+08:00 Shreyansh Jain <shreyansh.jain@nxp.com>:

> Hello Paul,
>
> Some comments inline, prepended with [SJ01]:
> (Would it be possible for you to send text mails to this mailing list - it
> helps in keeping replies contextual)
>
> From: Paul Tsvika [mailto:mozloverinweb@gmail.com]
> Sent: Tuesday, June 20, 2017 2:27 PM
> To: Shreyansh Jain <shreyansh.jain@nxp.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> page issues
>
> Hi Shreyansh,
>
> Thanks for the reply.
>
> By using the set-up script in DPDK:
>
> the hugepage memory looks to be allocated properly:
>
> cat /proc/meminfo
>
> HugePages_Total:     240
> HugePages_Free:     240
> HugePages_Rsvd:        0
> HugePages_Surp:         0
> Hugepagesize:       2048 kB
>
>
> and then i run the commands below in sequence:
>
> $ sudo modprobe uio
> $ sudo insmod ./build/kmod/igb_uio.ko
> $ sudo ./usertools/dpdk-devbind.py -b igb_uio xxx:xx.0 xxx:xx.1   (
> xxx.xx.0, xxx.xx.1 <-- 10G port pci address )
> $ sudo ./build/app/testpmd -l 1,2,3 -n 2 -- -i
>
> EAL: Detected 16 lcore(s)
> EAL: No free hugepages reported in hugepages-1048576kB
>
> [SJ01] DPDK iterates over all directories in /sys/kernel/mm/hugepages
> which contains one directory for each supported hugepage size. This log
> above is appearing because nothing was found for a directory named
> "hugepages-1048576kB". That is probably because you have '0' hugepages of
> size 1G in your system. This should not impact you until you are were
> expecting 1G hugepage size.
>
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:03:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:05:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:05:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
> socket=0
> Configuring Port 0 (socket 0)
> PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
> Port 0: 00:25:90:5C:E9:58
> Configuring Port 1 (socket 0)
> PMD: ixgbe_dev_link_status_print():  Port 1: Link Down
> Port 1: 00:25:90:5C:E9:59
> Checking link statuses...
> Done
> testpmd>
>
>
> Questions below:
>
> 1. It looks like the application can enter the interactive mode. however,
> I have no idea why EAL: No free hugepages reported in hugepages-1048576kB
> keeps popping out.
>
> [SJ01] Lets ignore that for while. You have about 240 pages of 2M =
> ~480Mb. Application is demanding ~339Mb (n=163456, size=2176)
>
>
> 2. Is there any reason why the ports went down why I run the commands? The
> Link was up before running it.
>
> [SJ01] This I am not sure. If the links were up _before_, they should be
> up now as well. Just out of curiosity, how did you check that links were up
> before? These links are not assigned to Linux kernel and are not visible in
> the ifconfig list.
>

*[P01] I don't know why you mentioned that these links are not assigned to
Linux kernel and not visible in the ifconfig list. **ixgbe is the inbox
driver of the kernel and eno3 and eno4 ( 10G ports ) appear in ifconfig
list.*
*However, eno3 and eno4 gone away ( disappeared in ifconfig ) when running
this command. I am still investigating the issue. *

>
> 3. The hugepage will be gone after running this application.
>
> HugePages_Total:     240
> HugePages_Free:         0
> HugePages_Rsvd:        0
> HugePages_Surp:         0
> Hugepagesize:       2048 kB
>
> It became this. However, how can i free it again ? I tried to umount and
> mount again but it did not work.
>
> [SJ01] Just go and delete all files created in /mnt/hugepages/ folder. You
> would have "HugePages_Free" available again. But, only if you have the
> application stopped.
>
>
> Please advice if any.
>
> [SJ01] Above is all what I know. I have no idea why your links are
> appearing down.
>
> -
> Shreyansh
>



Paul
-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Muhammad Zain-ul-Abideen]

HEy Paul,
Kindly select hugepages for NUMA architecture

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Shreyansh Jain]

Hi Paul,

Inline, with [SJ02]

From: Paul Tsvika [mailto:mozloverinweb@gmail.com] 
Sent: Wednesday, June 21, 2017 8:47 AM
To: Shreyansh Jain <shreyansh.jain@nxp.com>
Cc: users@dpdk.org
Subject: Re: [dpdk-users] Run testpmd application and encountered no free page issues

Thanks Shreyansh.

My comment prepended with [P01]


2017-06-20 19:10 GMT+08:00 Shreyansh Jain <mailto:shreyansh.jain@nxp.com>:
Hello Paul,

Some comments inline, prepended with [SJ01]:
(Would it be possible for you to send text mails to this mailing list - it helps in keeping replies contextual)

From: Paul Tsvika [mailto:mailto:mozloverinweb@gmail.com]
Sent: Tuesday, June 20, 2017 2:27 PM
To: Shreyansh Jain <mailto:shreyansh.jain@nxp.com>
Cc: mailto:users@dpdk.org
Subject: Re: [dpdk-users] Run testpmd application and encountered no free page issues

Hi Shreyansh,

Thanks for the reply.

By using the set-up script in DPDK:

the hugepage memory looks to be allocated properly:

cat /proc/meminfo

HugePages_Total:     240
HugePages_Free:     240
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB


and then i run the commands below in sequence:

$ sudo modprobe uio
$ sudo insmod ./build/kmod/igb_uio.ko
$ sudo ./usertools/dpdk-devbind.py -b igb_uio xxx:xx.0 xxx:xx.1   ( xxx.xx.0, xxx.xx.1 <-- 10G port pci address )
$ sudo ./build/app/testpmd -l 1,2,3 -n 2 -- -i

EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB

[SJ01] DPDK iterates over all directories in /sys/kernel/mm/hugepages which contains one directory for each supported hugepage size. This log above is appearing because nothing was found for a directory named "hugepages-1048576kB". That is probably because you have '0' hugepages of size 1G in your system. This should not impact you until you are were expecting 1G hugepage size.

EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:03:00.1 on NUMA socket 0
EAL:   probe driver: 8086:15ad net_ixgbe
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
Interactive-mode selected
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
Configuring Port 0 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
Port 0: 00:25:90:5C:E9:58
Configuring Port 1 (socket 0)
PMD: ixgbe_dev_link_status_print():  Port 1: Link Down
Port 1: 00:25:90:5C:E9:59
Checking link statuses...
Done
testpmd> 


Questions below:

1. It looks like the application can enter the interactive mode. however, I have no idea why EAL: No free hugepages reported in hugepages-1048576kB keeps popping out.

[SJ01] Lets ignore that for while. You have about 240 pages of 2M = ~480Mb. Application is demanding ~339Mb (n=163456, size=2176)


2. Is there any reason why the ports went down why I run the commands? The Link was up before running it.

[SJ01] This I am not sure. If the links were up _before_, they should be up now as well. Just out of curiosity, how did you check that links were up before? These links are not assigned to Linux kernel and are not visible in the ifconfig list.

[P01] I don't know why you mentioned that these links are not assigned to Linux kernel and not visible in the ifconfig list. ixgbe is the inbox driver of the kernel and eno3 and eno4 ( 10G ports ) appear in ifconfig list.
However, eno3 and eno4 gone away ( disappeared in ifconfig ) when running this command. I am still investigating the issue.  

[SJ02] That is not an issue. That is how it is supposed to work. When you start DPDK, it would unlink the devices from their standard ixgbe kernel module driver and link to uio/vfio driver for direct userspace interaction. That is the reason these links would not appear in ifconfig list – and that is the right way. Were you able to use these links when they were bound to ixgbe kernel module - that is, ping worked before DPDK was started? 



3. The hugepage will be gone after running this application.

HugePages_Total:     240
HugePages_Free:         0
HugePages_Rsvd:        0
HugePages_Surp:         0
Hugepagesize:       2048 kB

It became this. However, how can i free it again ? I tried to umount and mount again but it did not work.

[SJ01] Just go and delete all files created in /mnt/hugepages/ folder. You would have "HugePages_Free" available again. But, only if you have the application stopped.


Please advice if any.

[SJ01] Above is all what I know. I have no idea why your links are appearing down.

-
Shreyansh




Paul
-- 
P.T

Re: [dpdk-users] Run testpmd application and encountered no free page issues

[Paul Tsvika]

Hi Shreransh,

Inline [PT02],





2017-06-21 13:35 GMT+08:00 Shreyansh Jain <shreyansh.jain@nxp.com>:

> Hi Paul,
>
> Inline, with [SJ02]
>
> From: Paul Tsvika [mailto:mozloverinweb@gmail.com]
> Sent: Wednesday, June 21, 2017 8:47 AM
> To: Shreyansh Jain <shreyansh.jain@nxp.com>
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> page issues
>
> Thanks Shreyansh.
>
> My comment prepended with [P01]
>
>
> 2017-06-20 19:10 GMT+08:00 Shreyansh Jain <mailto:shreyansh.jain@nxp.com>:
> Hello Paul,
>
> Some comments inline, prepended with [SJ01]:
> (Would it be possible for you to send text mails to this mailing list - it
> helps in keeping replies contextual)
>
> From: Paul Tsvika [mailto:mailto:mozloverinweb@gmail.com]
> Sent: Tuesday, June 20, 2017 2:27 PM
> To: Shreyansh Jain <mailto:shreyansh.jain@nxp.com>
> Cc: mailto:users@dpdk.org
> Subject: Re: [dpdk-users] Run testpmd application and encountered no free
> page issues
>
> Hi Shreyansh,
>
> Thanks for the reply.
>
> By using the set-up script in DPDK:
>
> the hugepage memory looks to be allocated properly:
>
> cat /proc/meminfo
>
> HugePages_Total:     240
> HugePages_Free:     240
> HugePages_Rsvd:        0
> HugePages_Surp:         0
> Hugepagesize:       2048 kB
>
>
> and then i run the commands below in sequence:
>
> $ sudo modprobe uio
> $ sudo insmod ./build/kmod/igb_uio.ko
> $ sudo ./usertools/dpdk-devbind.py -b igb_uio xxx:xx.0 xxx:xx.1   (
> xxx.xx.0, xxx.xx.1 <-- 10G port pci address )
> $ sudo ./build/app/testpmd -l 1,2,3 -n 2 -- -i
>
> EAL: Detected 16 lcore(s)
> EAL: No free hugepages reported in hugepages-1048576kB
>
> [SJ01] DPDK iterates over all directories in /sys/kernel/mm/hugepages
> which contains one directory for each supported hugepage size. This log
> above is appearing because nothing was found for a directory named
> "hugepages-1048576kB". That is probably because you have '0' hugepages of
> size 1G in your system. This should not impact you until you are were
> expecting 1G hugepage size.
>
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:03:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:03:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:15ad net_ixgbe
> EAL: PCI device 0000:05:00.0 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> EAL: PCI device 0000:05:00.1 on NUMA socket 0
> EAL:   probe driver: 8086:1521 net_e1000_igb
> Interactive-mode selected
> USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
> socket=0
> Configuring Port 0 (socket 0)
> PMD: ixgbe_dev_link_status_print():  Port 0: Link Down
> Port 0: 00:25:90:5C:E9:58
> Configuring Port 1 (socket 0)
> PMD: ixgbe_dev_link_status_print():  Port 1: Link Down
> Port 1: 00:25:90:5C:E9:59
> Checking link statuses...
> Done
> testpmd>
>
>
> Questions below:
>
> 1. It looks like the application can enter the interactive mode. however,
> I have no idea why EAL: No free hugepages reported in hugepages-1048576kB
> keeps popping out.
>
> [SJ01] Lets ignore that for while. You have about 240 pages of 2M =
> ~480Mb. Application is demanding ~339Mb (n=163456, size=2176)
>
>
> 2. Is there any reason why the ports went down why I run the commands? The
> Link was up before running it.
>
> [SJ01] This I am not sure. If the links were up _before_, they should be
> up now as well. Just out of curiosity, how did you check that links were up
> before? These links are not assigned to Linux kernel and are not visible in
> the ifconfig list.
>
> [P01] I don't know why you mentioned that these links are not assigned to
> Linux kernel and not visible in the ifconfig list. ixgbe is the inbox
> driver of the kernel and eno3 and eno4 ( 10G ports ) appear in ifconfig
> list.
> However, eno3 and eno4 gone away ( disappeared in ifconfig ) when running
> this command. I am still investigating the issue.
>
> [SJ02] That is not an issue. That is how it is supposed to work. When you
> start DPDK, it would unlink the devices from their standard ixgbe kernel
> module driver and link to uio/vfio driver for direct userspace interaction.
> That is the reason these links would not appear in ifconfig list – and that
> is the right way. Were you able to use these links when they were bound to
> ixgbe kernel module - that is, ping worked before DPDK was started?
>
> [PT02]
>

You are correct. For some reasons that my 10G ports on board A did not work
even before running DPDK.
After changing a new board and run the steps mentioned above, it worked.
Thank you for the discussion in this mail thread.

>
> 3. The hugepage will be gone after running this application.
>
> HugePages_Total:     240
> HugePages_Free:         0
> HugePages_Rsvd:        0
> HugePages_Surp:         0
> Hugepagesize:       2048 kB
>
> It became this. However, how can i free it again ? I tried to umount and
> mount again but it did not work.
>
> [SJ01] Just go and delete all files created in /mnt/hugepages/ folder. You
> would have "HugePages_Free" available again. But, only if you have the
> application stopped.
>
>
> Please advice if any.
>
> [SJ01] Above is all what I know. I have no idea why your links are
> appearing down.
>
> -
> Shreyansh
>
>
>
>
> Paul
> --
> P.T
>



-- 
P.T