[dpdk-dev] client_server example application crash with virtio

[dpdk-dev] client_server example application crash with virtio

[Kyle Larose]

I just ran into an issue trying to run the client server example application using virito. Whenever the client (running as a secondary process) tried to send packets, it would crash. I traced the issue to an invalid vtpci_ops structure:

(gdb) p *vq->hw->vtpci_ops
$6 = {
  read_dev_cfg = 0x756d2073726f7470,
  write_dev_cfg = 0x6562206562207473,
  reset = 0x6425206e65657774,
  get_status = 0x20642520646e6120,
  set_status = 0x766973756c636e69,
  get_features = 0x7561666544202e65,
  set_features = 0xa5d64255b20746c,
  get_isr = 0x0,
  set_config_irq = 0x657472203a444d50,
  get_queue_num = 0x705f65626778635f,
  setup_queue = 0x203a7325203a646d,
  del_queue = 0x20746b702078616d,
  notify_queue = 0x7473756d206e656c
}

It looks like this is not being initialized in secondary processes, because we short-circuit the ethdev init here, before we call vtpci_init.

	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
		rx_func_get(eth_dev);
		return 0;
	}

Has  anyone submitted a patch to fix this?

I found that the following seemed to make it work, though I'm not sure it's appropriate. I can submit it as a patch if nobody else has done so yet. (I'd clean it up to return an error code if the init fails).

git diff drivers/net/virtio/virtio_ethdev.c
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index caa970c..5002847 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1002,6 +1002,8 @@ rx_func_get(struct rte_eth_dev *eth_dev)
                eth_dev->rx_pkt_burst = &virtio_recv_mergeable_pkts;
        else
                eth_dev->rx_pkt_burst = &virtio_recv_pkts;
+
+    vtpci_init(eth_dev->pci_dev, (struct virtio_hw*)eth_dev->data->dev_private);
 }

 /*

Thanks,

Kyle

Re: [dpdk-dev] client_server example application crash with virtio

[Kyle Larose]

Hrm. This may not be correct. It looks like there is some interaction between the primary and secondary here. vtpci_ops must be in shared memory. When I initialize my secondary, it overwrites the primary's with its address. I just checked, and without my patch, the secondary's vtpci_ops points to the location in the primary's address space:

In the primary:
EAL:   Not managed by a supported kernel driver, skipped
EAL: PCI Port IO found start=0xc060
>>>Initialized to ops: 0x59f6e0<<<<
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Device is blacklisted, not initializing

In the secondary:
(gdb) p vq->hw->vtpci_ops
$1 = (const struct virtio_pci_ops *) 0x59f6e0

It's not clear to me offhand how to resolve this. Unless the code to which the ops points is loaded at the exact same memory location, you need different ops pointers.

-----Original Message-----
From: Kyle Larose 
Sent: Friday, February 26, 2016 11:47 AM
To: 'dev@dpdk.org'
Subject: client_server example application crash with virtio

I just ran into an issue trying to run the client server example application using virito. Whenever the client (running as a secondary process) tried to send packets, it would crash. I traced the issue to an invalid vtpci_ops structure:

(gdb) p *vq->hw->vtpci_ops
$6 = {
  read_dev_cfg = 0x756d2073726f7470,
  write_dev_cfg = 0x6562206562207473,
  reset = 0x6425206e65657774,
  get_status = 0x20642520646e6120,
  set_status = 0x766973756c636e69,
  get_features = 0x7561666544202e65,
  set_features = 0xa5d64255b20746c,
  get_isr = 0x0,
  set_config_irq = 0x657472203a444d50,
  get_queue_num = 0x705f65626778635f,
  setup_queue = 0x203a7325203a646d,
  del_queue = 0x20746b702078616d,
  notify_queue = 0x7473756d206e656c
}

It looks like this is not being initialized in secondary processes, because we short-circuit the ethdev init here, before we call vtpci_init.

	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
		rx_func_get(eth_dev);
		return 0;
	}

Has  anyone submitted a patch to fix this?

I found that the following seemed to make it work, though I'm not sure it's appropriate. I can submit it as a patch if nobody else has done so yet. (I'd clean it up to return an error code if the init fails).

git diff drivers/net/virtio/virtio_ethdev.c
diff --git a/drivers/net/virtio/virtio_ethdev.c b/drivers/net/virtio/virtio_ethdev.c
index caa970c..5002847 100644
--- a/drivers/net/virtio/virtio_ethdev.c
+++ b/drivers/net/virtio/virtio_ethdev.c
@@ -1002,6 +1002,8 @@ rx_func_get(struct rte_eth_dev *eth_dev)
                eth_dev->rx_pkt_burst = &virtio_recv_mergeable_pkts;
        else
                eth_dev->rx_pkt_burst = &virtio_recv_pkts;
+
+    vtpci_init(eth_dev->pci_dev, (struct virtio_hw*)eth_dev->data->dev_private);
 }

 /*

Thanks,

Kyle