I am writing to consult about a technical issue with the Virtio network driver (located in driver/net/virtio). During our testing of the packed queue feature with desc_event_flags enabled, Testpmd consistently crashes after running for a short period. Below is a detailed description of the scenario, observation, root cause analysis, and a proposed fix！we hope to get clarification on whether this is a usage error or a potential driver issue.
- Test Environment: Running Testpmd with the Virtio PMD (
librte_pmd_virtio.so). - Test Focus: Validating the
desc_event_flags functionality of Virtio packed queue mode.
After Testpmd runs for several minutes, it crashes unexpectedly. Debugging shows that the content of vq_descx (within struct virtqueue) is being modified unexpectedly！likely due to address mismatches in the queue memory layout.
testpmd -c 0xff -n 4 --huge-dir=/mnt/huge_dpdk --socket-mem=1024 --socket-limit=1024 -w 0000:15:00.1 --file-prefix=test3 -d /usr/lib64/librte_pmd_virtio.so \
-- --total-num-mbufs=115200 --rxq=4 --txq=4 --forward-mode=txonly --nb-cores=4 --stats-period 1 --burst=512 --rxd=512 --txd=512 --eth-peer=0,10:70:fd:2a:60:39
The Virtio driver uses inconsistent address calculation logic for two critical steps:
- When informing the hardware (via the
modern_setup_queue interface) of the physical addresses for the driver and device regions of the packed queue. - When calculating the virtual addresses of the
driver and device regions for the driver¨s own use (via vring_init_packed).
This mismatch leads to the hardware and the driver referencing different memory regions for the device queue, causing unintended overwrites of vq_descx.
The modern_setup_queue function configures the queue addresses and passes them to the hardware. For packed queues, it calculates desc_addr, avail_addr (driver region), and used_addr (device region) as follows:
Key parameters for our test:
vq_nentries = 0x1000 (4096 entries)sizeof(struct vring_desc) = 16 (0x10 in hex)offsetof(struct vring_avail, ring[vq->vq_nentries]) = 4 + (0x1000 * 2) (base offset 4 + 2 bytes per ring entry)- Alignment requirement:
VIRTIO_PCI_VRING_ALIGN = 4096 (0x1000 in hex)
Example calculation (assuming desc_addr = 0x0):
avail_addr = 0x0 + (0x1000 * 0x10) = 0x10000 (driver region address passed to hardware)used_addr = RTE_ALIGN_CEIL(0x10000 + 4 + (0x1000 * 2), 0x1000) = RTE_ALIGN_CEIL(0x12004, 0x1000) = 0x13000 (device region address passed to hardware)
The vring_init_packed function calculates the driver-internal virtual addresses for the packed queue¨s driver and device regions:
Example calculation (same desc_addr = 0x0, p = 0x0):
vr->driver = 0x0 + (0x1000 * 0x10) = 0x10000 (matches avail_addr from modern_setup_queue)vr->device = RTE_ALIGN_CEIL(0x10000 + sizeof(struct vring_packed_desc_event), 0x1000)- Assuming
sizeof(struct vring_packed_desc_event) = 4 (standard definition), this becomes RTE_ALIGN_CEIL(0x10004, 0x1000) = 0x11000 (driver-internal device region address)
- Hardware uses
0x13000 as the device region address. - Driver uses
0x11000 as the device region address.
We modified modern_setup_queue to use the same address logic as vring_init_packed for packed queues. After this change, Testpmd runs stably without crashes:
We noticed that the virtio-user and vhost-user drivers already use the same logic as our modified modern_setup_queue for packed queues. For example, in virtio_user_setup_queue_packed:
Therefore, for packed queues, why do modern_setup_queue and vring_init_packed use different logic to calculate the device region address?
Is this inconsistency due to incorrect usage on my part, or are there special considerations specific to packed queue mode (e.g., hardware compatibility, protocol requirements)?
We would greatly appreciate your help in clarifying this confusion. Thank you!
Best regards.
[A DPDK user and developer].
