[dpdk-dev] i40e and memory allocations restricted on node 1

[dpdk-dev] i40e and memory allocations restricted on node 1

[Ivan Nardi]

Hi
With dpdk 17.02-rc2 i40e driver doesn't load at all when memory
allocation is restricted on numa node 1 (on system with more than 1
node, obviously)


[root@micro ~]# /tmp/testpmd -d /home/micro/lib/librte_pmd_i40e.so -c
0xFFFC000 -w 0000:81:00.0  -w 0000:81:00.1 -n 4 --socket-mem=0,8192 --
-i --socket-num=1
EAL: Detected 56 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:1572 net_i40e
PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
RING: Cannot reserve memory
HASH: memory allocation failed
PMD: i40e_init_ethtype_filter_list(): Failed to create ethertype hash table!
EAL: Error - exiting with code: 1
  Cause: Requested device 0000:81:00.0 cannot be used


Everything is fine with 16.11, or allowing allocations from both nodes

I was able to locate the code to fix (at least, I hope so), but I
don't really know if it is better to allocate from SOCKET_ID_ANY of
from the exact node associated with the network device, something like
dev->data->numa_node ("pseudo" patch below)


--- dpdk-17.02-rc2.orig/drivers/net/i40e/i40e_ethdev.c    2017-01-30
23:47:11.000000000 +0100
+++ dpdk-17.02-rc2/drivers/net/i40e/i40e_ethdev.c    2017-02-06
21:53:37.812313120 +0100
@@ -899,6 +899,7 @@
         .entries = I40E_MAX_ETHERTYPE_FILTER_NUM,
         .key_len = sizeof(struct i40e_ethertype_filter_input),
         .hash_func = rte_hash_crc,
+        .socket_id = SOCKET_ID_ANY,
     };

     /* Initialize ethertype filter rule list and hash */
@@ -942,6 +943,7 @@
         .entries = I40E_MAX_TUNNEL_FILTER_NUM,
         .key_len = sizeof(struct i40e_tunnel_filter_input),
         .hash_func = rte_hash_crc,
+        .socket_id = SOCKET_ID_ANY,
     };

     /* Initialize tunnel filter rule list and hash */
@@ -985,6 +987,7 @@
         .entries = I40E_MAX_FDIR_FILTER_NUM,
         .key_len = sizeof(struct rte_eth_fdir_input),
         .hash_func = rte_hash_crc,
+        .socket_id = SOCKET_ID_ANY,
     };

     /* Initialize flow director filter rule list and hash */



Any thoughts?
Thanks in advance
Ivan

Re: [dpdk-dev] i40e and memory allocations restricted on node 1

[Zhang, Helin]

> -----Original Message-----
> From: Ivan Nardi [mailto:nardi.ivan@gmail.com]
> Sent: Tuesday, February 7, 2017 5:04 AM
> To: dev@dpdk.org
> Cc: Zhang, Helin; Wu, Jingjing
> Subject: i40e and memory allocations restricted on node 1
> 
> Hi
> With dpdk 17.02-rc2 i40e driver doesn't load at all when memory allocation is
> restricted on numa node 1 (on system with more than 1 node, obviously)
> 
> 
> [root@micro ~]# /tmp/testpmd -d /home/micro/lib/librte_pmd_i40e.so -c
> 0xFFFC000 -w 0000:81:00.0  -w 0000:81:00.1 -n 4 --socket-mem=0,8192 -- -i --
> socket-num=1
> EAL: Detected 56 lcore(s)
> EAL: Probing VFIO support...
> EAL: PCI device 0000:81:00.0 on NUMA socket 1
> EAL:   probe driver: 8086:1572 net_i40e
> PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
This is a very old verion of firmware, it may have compatibility issue.
So first, firmware upgrade is needed.

> RING: Cannot reserve memory
It seems not reserve memory for ring, and not an issue specifically to i40e.
Let's double check that if it is common to all.

> HASH: memory allocation failed
> PMD: i40e_init_ethtype_filter_list(): Failed to create ethertype hash table!
> EAL: Error - exiting with code: 1
>   Cause: Requested device 0000:81:00.0 cannot be used
> 
> 
> Everything is fine with 16.11, or allowing allocations from both nodes
> 
> I was able to locate the code to fix (at least, I hope so), but I don't really
> know if it is better to allocate from SOCKET_ID_ANY of from the exact node
> associated with the network device, something like
> dev->data->numa_node ("pseudo" patch below)
> 
> 
> --- dpdk-17.02-rc2.orig/drivers/net/i40e/i40e_ethdev.c    2017-01-30
> 23:47:11.000000000 +0100
> +++ dpdk-17.02-rc2/drivers/net/i40e/i40e_ethdev.c    2017-02-06
> 21:53:37.812313120 +0100
> @@ -899,6 +899,7 @@
>          .entries = I40E_MAX_ETHERTYPE_FILTER_NUM,
>          .key_len = sizeof(struct i40e_ethertype_filter_input),
>          .hash_func = rte_hash_crc,
> +        .socket_id = SOCKET_ID_ANY,
>      };
> 
>      /* Initialize ethertype filter rule list and hash */ @@ -942,6 +943,7 @@
>          .entries = I40E_MAX_TUNNEL_FILTER_NUM,
>          .key_len = sizeof(struct i40e_tunnel_filter_input),
>          .hash_func = rte_hash_crc,
> +        .socket_id = SOCKET_ID_ANY,
>      };
> 
>      /* Initialize tunnel filter rule list and hash */ @@ -985,6 +987,7 @@
>          .entries = I40E_MAX_FDIR_FILTER_NUM,
>          .key_len = sizeof(struct rte_eth_fdir_input),
>          .hash_func = rte_hash_crc,
> +        .socket_id = SOCKET_ID_ANY,
>      };
> 
>      /* Initialize flow director filter rule list and hash */
> 
> 
> 
> Any thoughts?
> Thanks in advance
> Ivan

Re: [dpdk-dev] i40e and memory allocations restricted on node 1

[Ivan Nardi]

Hi Helin
no luck with the latest nvm

PMD: eth_i40e_dev_init(): FW 5.0 API 1.5 NVM 05.00.05 eetrack 800028ac
RING: Cannot reserve memory
HASH: memory allocation failed
PMD: i40e_init_ethtype_filter_list(): Failed to create ethertype hash table!
EAL: Error - exiting with code: 1

It seems to me the issue is that i40e code try to allocate hash tables
(for tunnel filter list, fdir filter list and ethtype filter list)
from node 0, because it doesn't explicitly set the field socket_id in
struct rte_hash_parameters.
If my analysis is correct, I am wondering if it should allocate memory
from SOCKET_ID_ANY of from the exact node associated with the network
device

Thanks for looking into that

Ivan



On 7 February 2017 at 02:49, Zhang, Helin <helin.zhang@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: Ivan Nardi [mailto:nardi.ivan@gmail.com]
>> Sent: Tuesday, February 7, 2017 5:04 AM
>> To: dev@dpdk.org
>> Cc: Zhang, Helin; Wu, Jingjing
>> Subject: i40e and memory allocations restricted on node 1
>>
>> Hi
>> With dpdk 17.02-rc2 i40e driver doesn't load at all when memory allocation is
>> restricted on numa node 1 (on system with more than 1 node, obviously)
>>
>>
>> [root@micro ~]# /tmp/testpmd -d /home/micro/lib/librte_pmd_i40e.so -c
>> 0xFFFC000 -w 0000:81:00.0  -w 0000:81:00.1 -n 4 --socket-mem=0,8192 -- -i --
>> socket-num=1
>> EAL: Detected 56 lcore(s)
>> EAL: Probing VFIO support...
>> EAL: PCI device 0000:81:00.0 on NUMA socket 1
>> EAL:   probe driver: 8086:1572 net_i40e
>> PMD: eth_i40e_dev_init(): FW 4.40 API 1.4 NVM 04.05.03 eetrack 80001cd8
> This is a very old verion of firmware, it may have compatibility issue.
> So first, firmware upgrade is needed.
>
>> RING: Cannot reserve memory
> It seems not reserve memory for ring, and not an issue specifically to i40e.
> Let's double check that if it is common to all.
>
>> HASH: memory allocation failed
>> PMD: i40e_init_ethtype_filter_list(): Failed to create ethertype hash table!
>> EAL: Error - exiting with code: 1
>>   Cause: Requested device 0000:81:00.0 cannot be used
>>
>>
>> Everything is fine with 16.11, or allowing allocations from both nodes
>>
>> I was able to locate the code to fix (at least, I hope so), but I don't really
>> know if it is better to allocate from SOCKET_ID_ANY of from the exact node
>> associated with the network device, something like
>> dev->data->numa_node ("pseudo" patch below)
>>
>>
>> --- dpdk-17.02-rc2.orig/drivers/net/i40e/i40e_ethdev.c    2017-01-30
>> 23:47:11.000000000 +0100
>> +++ dpdk-17.02-rc2/drivers/net/i40e/i40e_ethdev.c    2017-02-06
>> 21:53:37.812313120 +0100
>> @@ -899,6 +899,7 @@
>>          .entries = I40E_MAX_ETHERTYPE_FILTER_NUM,
>>          .key_len = sizeof(struct i40e_ethertype_filter_input),
>>          .hash_func = rte_hash_crc,
>> +        .socket_id = SOCKET_ID_ANY,
>>      };
>>
>>      /* Initialize ethertype filter rule list and hash */ @@ -942,6 +943,7 @@
>>          .entries = I40E_MAX_TUNNEL_FILTER_NUM,
>>          .key_len = sizeof(struct i40e_tunnel_filter_input),
>>          .hash_func = rte_hash_crc,
>> +        .socket_id = SOCKET_ID_ANY,
>>      };
>>
>>      /* Initialize tunnel filter rule list and hash */ @@ -985,6 +987,7 @@
>>          .entries = I40E_MAX_FDIR_FILTER_NUM,
>>          .key_len = sizeof(struct rte_eth_fdir_input),
>>          .hash_func = rte_hash_crc,
>> +        .socket_id = SOCKET_ID_ANY,
>>      };
>>
>>      /* Initialize flow director filter rule list and hash */
>>
>>
>>
>> Any thoughts?
>> Thanks in advance
>> Ivan