Hello,I'm developing a PMD for a NIC that I'm developing at the moment.
(DPDK version 20.11)
The NIC should forward packets between a network device (based on dpaa2 HW) and a PCIe device.
I'm running l2fwd application in order to develop the PMD, with the following command:
./dpdk-l2fwd -c 0x3 --vdev=my_pmd0 -- -p 0x3 -T 0
The application uses 2 lcores:
- Lcore 0: RX port 0 TX port 1.
- Lcore 1: RX port 1 TX port 0.
Where port 0 is the physical one, and net_dpaa2 PMD is attached to this port.
Port 1 is the virtual one (pci device), and my PMD is attached to it.
The problem I'm experiencing is related to the tx_pkt_burst callback.
When the tx_pkt_burst callback is called, I start the DMA operations to write the packets, but the problem is that I need to wait for the DMA operations to finish in order to do some more work.
I don't want to lock the lcore in my tx_pkt_burst function waiting for the DMA operation to finish, I want to release it as soon as possible, so it will go back polling the physical port.
Another issue is that I need to poll the PCI for new available write buffers.
I would like to handle all the TX related software work using one core, including polling for new write buffers, starting the DMA operations once new packets arrive and polling DMA until the operation finishes, then doing some more software work.
Is there a "correct" way to handle this?
What I've tried:
tx_pkt_burst just stores the new packets in an array, and rx_pkt_burst handles all the TX tasks (polling for new write buffers, Starts DMA operations, and polls the DMA operations status).
This works, but:
- The workload is not symmetric, the RX lcore does all the "heavy lifting", while the TX lcore does almost nothing.
- spinlocks are needed since 2 lcores access the same data.
- Less efficient caches.
What I've though:
- Using rte_eal_alarm_set to poll the tx_pkt_burst function, for example:
uint16_t my_tx_pkt_burst(void *txq, struct rte_mbuf **tx_pkts,
uint16_t nb_pkts)
{
/* Save mbufs in an array and return number of saved mbufs */
}
void tx_poll(void *param)
{
/* Check for new write buffers*/
/* Start DMA operations for new mbufs from my_tx_pkt_burst */
/* Check DMA status for prev. operations, and do
* some more work if needed
*/
/* Call this function again in 1us */
rte_eal_alarm_set(1, tx_poll, param);
}