[dpdk-dev] rte_eth_tx_queue_setup() failing (error -22) when setting up tx queues on a VMXNET3 port...

[dpdk-dev] rte_eth_tx_queue_setup() failing (error -22) when setting up tx queues on a VMXNET3 port...

[Montorsi, Francesco]

Hi all,
I have a server running VMWare ESXi 5.5.0 and VMWare vCenter 5.5.0. On such server I created a VM with two VMXNET3 NIC cards (one for management, the other one should be used with DPDK to enable fast-RX of packets coming from other VMs / bare-metal NICs).

Inside the VM I have successfully created 2MB hugepages, loaded igb_uio driver and binded the 2nd VMXNET3 NIC to igb_uio driver:

----------------------------------------------------------------------------------------------------------------------
$ ./dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3 unused=igb_uio *Active*

Other network devices
=====================
<none>
----------------------------------------------------------------------------------------------------------------------

However when I start my DPDK-application I get the following error during TX queue initialization:

ERR rte_eth_tx_queue_setup: err=-22, port=0: Unknown error -22

This is the code that I'm using:

              ...
               rte_eth_dev_info_get(m_portid, m_dev_info);


	// proceed with configuration

	struct rte_eth_conf port_conf;
	memset(&port_conf, 0, sizeof(port_conf));

	port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
		/** The multi-queue packet distribution mode to be used, e.g. RSS.; this is important to use multiple RX queues per port ID */
	port_conf.rxmode.max_rx_pkt_len = ETHER_MAX_LEN;
	port_conf.rxmode.hw_strip_crc   = 1; /**< enable CRC stripping by hardware */
	port_conf.rxmode.jumbo_frame    = 1; /**< Jumbo Frame Support enabled */
	//port_conf.rxmode.enable_lro     = 1; /**< Enable LRO */		// NOT SUPPORTED ON TESTED HW
	port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP;
	port_conf.txmode.mq_mode = ETH_MQ_TX_NONE;

	m_num_queues = MIN(64, m_dev_info->max_rx_queues);
	m_num_queues = MIN(128, m_num_queues);
	ret = rte_eth_dev_configure(m_portid, m_num_queues, 1 /* number of tx queues */, &port_conf);
	if (ret < 0)
	{
		HMLogError("HwEmulDPDKPort::init() rte_eth_dev_configure: err=%d, port=%u: %s", ret, m_portid, rte_strerror(ret));
		return false;
	}

	// init one TX queue: even if we never ever TX packets, at least 1 queue is needed!
--->	ret = rte_eth_tx_queue_setup(m_portid, 0 /* queue ID */, 64 /* num descriptors */, rte_eth_dev_socket_id(m_portid), NULL);
	if (ret < 0)
	{
		// retry with just 1 descriptor

--->		ret = rte_eth_tx_queue_setup(m_portid, 0 /* queue ID */, 1 /* num descriptors */, rte_eth_dev_socket_id(m_portid), NULL);
		if (ret < 0)
		{
			HMLogError("HwEmulDPDKPort::init() rte_eth_tx_queue_setup: err=%d, port=%u: %s", ret, m_portid, rte_strerror(ret));
			return false;
		}
	}


Basically since I want to only receive packets (no TX) I'm not really interested in TX queues / num of TX descriptors... However I have troubles decrypting this -22 error code... any hint?


Thanks a lot,

Francesco Montorsi

Re: [dpdk-dev] rte_eth_tx_queue_setup() failing (error -22) when setting up tx queues on a VMXNET3 port...

[Stephen Hemminger]

On Wed, 25 Nov 2015 10:32:33 +0000
"Montorsi, Francesco" <fmontorsi@empirix.com> wrote:

> Hi all,
> I have a server running VMWare ESXi 5.5.0 and VMWare vCenter 5.5.0. On such server I created a VM with two VMXNET3 NIC cards (one for management, the other one should be used with DPDK to enable fast-RX of packets coming from other VMs / bare-metal NICs).
> 
> Inside the VM I have successfully created 2MB hugepages, loaded igb_uio driver and binded the 2nd VMXNET3 NIC to igb_uio driver:
> 
> ----------------------------------------------------------------------------------------------------------------------
> $ ./dpdk_nic_bind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=igb_uio unused=vmxnet3
> 
> Network devices using kernel driver
> ===================================
> 0000:03:00.0 'VMXNET3 Ethernet Controller' if=eth0 drv=vmxnet3 unused=igb_uio *Active*
> 
> Other network devices
> =====================
> <none>
> ----------------------------------------------------------------------------------------------------------------------
> 
> However when I start my DPDK-application I get the following error during TX queue initialization:
> 
> ERR rte_eth_tx_queue_setup: err=-22, port=0: Unknown error -22
> 
> This is the code that I'm using:
> 
>               ...
>                rte_eth_dev_info_get(m_portid, m_dev_info);
> 
> 
> 	// proceed with configuration
> 
> 	struct rte_eth_conf port_conf;
> 	memset(&port_conf, 0, sizeof(port_conf));
> 
> 	port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> 		/** The multi-queue packet distribution mode to be used, e.g. RSS.; this is important to use multiple RX queues per port ID */
> 	port_conf.rxmode.max_rx_pkt_len = ETHER_MAX_LEN;
> 	port_conf.rxmode.hw_strip_crc   = 1; /**< enable CRC stripping by hardware */
> 	port_conf.rxmode.jumbo_frame    = 1; /**< Jumbo Frame Support enabled */
> 	//port_conf.rxmode.enable_lro     = 1; /**< Enable LRO */		// NOT SUPPORTED ON TESTED HW
> 	port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP;
> 	port_conf.txmode.mq_mode = ETH_MQ_TX_NONE;
> 
> 	m_num_queues = MIN(64, m_dev_info->max_rx_queues);
> 	m_num_queues = MIN(128, m_num_queues);
> 	ret = rte_eth_dev_configure(m_portid, m_num_queues, 1 /* number of tx queues */, &port_conf);
> 	if (ret < 0)
> 	{
> 		HMLogError("HwEmulDPDKPort::init() rte_eth_dev_configure: err=%d, port=%u: %s", ret, m_portid, rte_strerror(ret));
> 		return false;
> 	}
> 
> 	// init one TX queue: even if we never ever TX packets, at least 1 queue is needed!
> --->	ret = rte_eth_tx_queue_setup(m_portid, 0 /* queue ID */, 64 /* num descriptors */, rte_eth_dev_socket_id(m_portid), NULL);
> 	if (ret < 0)
> 	{
> 		// retry with just 1 descriptor
> 
> --->		ret = rte_eth_tx_queue_setup(m_portid, 0 /* queue ID */, 1 /* num descriptors */, rte_eth_dev_socket_id(m_portid), NULL);
> 		if (ret < 0)
> 		{
> 			HMLogError("HwEmulDPDKPort::init() rte_eth_tx_queue_setup: err=%d, port=%u: %s", ret, m_portid, rte_strerror(ret));
> 			return false;
> 		}
> 	}
> 
> 
> Basically since I want to only receive packets (no TX) I'm not really interested in TX queues / num of TX descriptors... However I have troubles decrypting this -22 error code... any hint?

First advice, DPDK is open source, therefore you have the source use it.

Error codes match Linux/Unix errno's. You can use strerror(-ret) to find the value
or look in errno.h. Hint -22 == EINVAL

If you read the source, you will see that it there are log messages enabled if you
configure with LIBRTE_ETHDEV_DEBUG enabled and rebuild DPDK. And there are log messages
for VMXNETE3 controlled by LIBRTE_VMXNET3_DEBUG_INIT

Re: [dpdk-dev] rte_eth_tx_queue_setup() failing (error -22) when setting up tx queues on a VMXNET3 port...

[Montorsi, Francesco]

Hi Stephen,

> -----Original Message-----
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> 
> If you read the source, you will see that it there are log messages enabled if
> you configure with LIBRTE_ETHDEV_DEBUG enabled and rebuild DPDK. And
> there are log messages for VMXNETE3 controlled by
> LIBRTE_VMXNET3_DEBUG_INIT

Thanks for this hint. I didn't know actually where to read... anyway enabling these additional debug messages I found the cause of the problem: the minimal TX queue ring size for VMXNET3 is 512; I was passing 64.
Fixed this problem, I'm now on the next one.... I discovered that apparently the driver for VMXNET3 supports only a limited set of functions:

(gdb) p *dev->dev_ops
$3 = {dev_configure = 0x75cda5 <vmxnet3_dev_configure>, dev_start = 0x75d653 <vmxnet3_dev_start>, dev_stop = 0x75d799 <vmxnet3_dev_stop>, dev_set_link_up = 0, dev_set_link_down = 0,
  dev_close = 0x75d8ab <vmxnet3_dev_close>, promiscuous_enable = 0x75dd12 <vmxnet3_dev_promiscuous_enable>, promiscuous_disable = 0x75ddf6 <vmxnet3_dev_promiscuous_disable>,
  allmulticast_enable = 0x75df0c <vmxnet3_dev_allmulticast_enable>, allmulticast_disable = 0x75df40 <vmxnet3_dev_allmulticast_disable>, link_update = 0x75dbd9 <vmxnet3_dev_link_update>,
  stats_get = 0x75d8fe <vmxnet3_dev_stats_get>, stats_reset = 0, xstats_get = 0, xstats_reset = 0, queue_stats_mapping_set = 0, dev_infos_get = 0x75db7f <vmxnet3_dev_info_get>,
  mtu_set = 0, vlan_filter_set = 0x75df74 <vmxnet3_dev_vlan_filter_set>, vlan_tpid_set = 0, vlan_strip_queue_set = 0, vlan_offload_set = 0x75e455 <vmxnet3_dev_vlan_offload_set>,
  vlan_pvid_set = 0, rx_queue_start = 0, rx_queue_stop = 0, tx_queue_start = 0, tx_queue_stop = 0, rx_queue_setup = 0x75bfb8 <vmxnet3_dev_rx_queue_setup>,
  rx_queue_release = 0x74b069 <vmxnet3_dev_rx_queue_release>, rx_queue_count = 0, rx_descriptor_done = 0, rx_queue_intr_enable = 0, rx_queue_intr_disable = 0,
  tx_queue_setup = 0x75bbc6 <vmxnet3_dev_tx_queue_setup>, tx_queue_release = 0x74b03c <vmxnet3_dev_tx_queue_release>, dev_led_on = 0, dev_led_off = 0, flow_ctrl_get = 0,
  flow_ctrl_set = 0, priority_flow_ctrl_set = 0, mac_addr_remove = 0, mac_addr_add = 0, mac_addr_set = 0, uc_hash_table_set = 0, uc_all_hash_table_set = 0, mirror_rule_set = 0,
  mirror_rule_reset = 0, set_vf_rx_mode = 0, set_vf_rx = 0, set_vf_tx = 0, set_vf_vlan_filter = 0, udp_tunnel_add = 0, udp_tunnel_del = 0, set_queue_rate_limit = 0, set_vf_rate_limit = 0,
  fdir_add_signature_filter = 0, fdir_update_signature_filter = 0, fdir_remove_signature_filter = 0, fdir_infos_get = 0, fdir_add_perfect_filter = 0, fdir_update_perfect_filter = 0,
  fdir_remove_perfect_filter = 0, fdir_set_masks = 0, reta_update = 0, reta_query = 0, get_reg_length = 0, get_reg = 0, get_eeprom_length = 0, get_eeprom = 0, set_eeprom = 0,
  rss_hash_update = 0, rss_hash_conf_get = 0, filter_ctrl = 0, set_mc_addr_list = 0, timesync_enable = 0, timesync_disable = 0, timesync_read_rx_timestamp = 0,
  timesync_read_tx_timestamp = 0}


Since I was using the rte_eth_rx_queue_count() function in one place, and VMXNET3 does not support it, I'm getting a SEGFAULT.
So next question is:  is user's task to check for validity of pointers inside dev_ops before calling driver functions? Because rte_eth_rx_queue_count() and companion funcitons have no safety checks apparently (!!!)

Thanks!

Francesco

Re: [dpdk-dev] rte_eth_tx_queue_setup() failing (error -22) when setting up tx queues on a VMXNET3 port...

[Thomas Monjalon]

2015-11-25 17:55, Montorsi, Francesco:
> Since I was using the rte_eth_rx_queue_count() function in one place, and VMXNET3 does not support it, I'm getting a SEGFAULT.
> So next question is:  is user's task to check for validity of pointers inside dev_ops before calling driver functions? Because rte_eth_rx_queue_count() and companion funcitons have no safety checks apparently (!!!)

Bruce already sent a fix for this issue.
It should be applied on HEAD in few minutes.