Hi JJ, Sorry for the late reply. Here are the answers to your questions. Still having the same issue. Nothing is reaching to DPDK application after ping . I tried to run l2fwd and found the same issue on VM2 after running l2fwd I have two Mac addresses for port 0 : FA:16:3E:EF:2C:26 and Port 1, MAC address: FA:16:3E:0D:39:EC then I added entry in to ARP table to associate IP address with these mac address on VM2 like : arp -s x.y.t.z FA:16:3E:EF:2C:26 dev ens3(ens3 is bind with kernel module) arp -s x.y.t.r FA:16:3E:0D:39:EC dev ens3 After it I tried to ping x.y.t.z assuming that this message will go to port 0 RX queue but not see any message in l2fwd . Here is a log snapshot: "iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 done: Port 0, MAC address: FA:16:3E:EF:2C:26 Initializing port 1... iavf_dev_init_vlan(): Failed to update vlan offload iavf_dev_configure(): configure VLAN failed: -95 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 25 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26 iavf_dev_rx_queue_setup(): >> iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0. check_rx_vec_allow(): Vector Rx can be enabled on this rxq. iavf_dev_tx_queue_setup(): >> check_tx_vec_allow(): Vector tx can be enabled on this txq. iavf_dev_start(): >> iavf_set_rx_function(): request RXDID[22] in Queue[0] is not supported, set rx_pkt_burst as legacy for all queues iavf_set_rx_function(): Using avx2 Vector Rx (port 1). iavf_set_tx_function(): Using avx2 Vector Tx (port 1). iavf_configure_queues(): RXDID[22] is not supported, request default RXDID[1] in Queue[0] iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7 iavf_add_del_all_mac_addr(): add/rm mac:FA:16:3E:0D:39:EC iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10 iavf_dev_tx_queue_start(): >> iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event iavf_dev_rx_queue_start(): >> iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_dev_alarm_handler(): ICR01_ADMINQ is reported iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 done: Port 1, MAC address: FA:16:3E:0D:39:EC Skipping disabled port 2 Checking link statusdone Port 0 Link up at 25 Gbps FDX Autoneg Port 1 Link up at 25 Gbps FDX Autoneg L2FWD: entering main loop on lcore 2 L2FWD: -- lcoreid=2 portid=1 L2FWD: entering main loop on lcore 1 L2FWD: -- lcoreid=1 portid=0" and then log I added at RX queue to dump source and dest in Rx buffer: L2FWD: ITE one ----nb_rx=1 portid=0 L2FWD: ITR ---2----src=134610945 dest=134219268 I added this login /* main processing loop */ static void l2fwd_main_loop(void) { in /* Read packet from RX queues. 8< */ second loop for (i = 0; i < qconf->n_rx_port; i++) { portid = qconf->rx_port_list[i]; nb_rx = rte_eth_rx_burst(portid, 0, pkts_burst, MAX_PKT_BURST); port_statistics[portid].rx += nb_rx; for (j = 0; j < nb_rx; j++) { m = pkts_burst[j]; uint32_t ip_dst,ip_src; struct rte_ipv4_hdr *ip_hdr; rte_prefetch0(rte_pktmbuf_mtod(m, void *)); ip_hdr = rte_pktmbuf_mtod(m, struct rte_ipv4_hdr *); ip_dst = rte_be_to_cpu_32(ip_hdr->dst_addr); ip_src = rte_be_to_cpu_32(ip_hdr->src_addr); RTE_LOG(INFO, L2FWD, "ITE one ----nb_rx=%u portid=%u\n", nb_rx,portid); RTE_LOG(INFO, L2FWD, "ITR ---2----src=%u dest=%u\n",ip_src,ip_dst); l2fwd_simple_forward(m, portid); } } Please let me know why the ping message is not reaching dpdk even though on the source side Thanks, Dinesh On Tue, Apr 18, 2023 at 3:51 AM J.J. Martzki wrote: > From your picture, l3fwd is running in VM2, with a tap vdev. But > what's the relationship between VM1 and VM2? Are they running on the > same host? > Yes, They are running on the same host with 3 NICs two are bind with dpdk and one is with kernel module, These NICs are on separate subnets > > I assume: > 1. VM1 and VM2 are running on the same host. > 2. ens05 is a net device in VM1. > 3. VM1 has its own net device, let's call it ens06, and IP2 is on it. > Yes: > 0000:00:05.0 'Ethernet Virtual Function 700 Series 154c' drv=uio_pci_generic unused=iavf,vfio-pci 0000:00:06.0 'Ethernet Virtual Function 700 Series 154c' drv=uio_pci_generic unused=iavf,vfio-pci Network devices using kernel driver =================================== 0000:00:03.0 'Virtio network device 1000' if=ens3 drv=virtio-pci unused=vfio-pci,uio_pci_generic *Active* > Based on the assumptions above, you should make sure VM1 and VM2 can > communicate with each other without l3fwd, just through the linux > network stack. > Yes . I tested with ens05 and ens03 and ping is working fine without dpdk. > This ensures the link between these 2 VMs is good (ens05 in VM2 and > ens06 in VM1 can reach each other directly). Maybe they are connected > by a linux bridge on the host or something like that. > > And you l3fwd should attach to ens05, by vfio or uio. Just use > dpdk-devbind.py to do this. > After this, ens05 is no longer available in the kernel, and l3fwd can > use it. As your command line '-p 0x3' described, l3fwd will use port 0 > and port 1. > You can use l3fwd init log to make sure there are 2 ports available > for l3fwd as we expected. > > BTW, I suggest you start from test-pmd first. test-pmd can connect > ports ethier. But it supports interactive mode and many command lines. > Such as display of port info like MAC you mentioned before. > > > Please share some link that has details descriptions about testing this > flow : > I didn't find a suitable article to describe this, maybe you can try: > > https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk > In this article, the DPDK application is ovs, just like your l3fwd. > And it can attach many ports not only vhost ports but also physical > ports, vdev ports like tap and so on. > > J.J. Martzki, > Best wishes. > > > Dinesh Kumar 于2023年4月17日周一 23:47写道： > > > > Hi JJ, > > Thank you for the clarification. I am new to this so trying to > understand how it will work. > > > > Please share some link that has details descriptions about testing this > flow : > > > > | VM1 | HOST > > | VM2 | > > app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app > > > > Sorry for my misunderstanding. > > > > here is the correct flow I am working on now : > > After running the below command from VM2, I am pinging IP address of > VM1 vi network namespace ns0, but not able to see any message in l3fwd and > ping is unreachable. > > Thanks for your time and any help will be really appreciated. > > > > > > > > On Fri, Apr 14, 2023 at 10:46 PM J.J. Martzki > wrote: > >> > >> > Is this process correct to assign IP and then device to DPDK? > >> No, after binding the driver to vfio/uio, the IP configuration is no > >> longer available. > >> The IP is something belonging to an application, not a driver (DPDK). > >> > >> > Is there any way to see ip and ethernet address after binding to DPDK? > >> MAC is available for the rte_eth_dev, there's some API like > >> rte_eth_macaddr_get() to get MAC > >> from a specific port. l3fwd should have some CLI commands for getting > >> MAC ethier, maybe > >> you can read it's documents. > >> IP is not available as I said before. > >> > >> BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate > >> VMs? It's not gonna work. > >> AFAK, l3fwd switches packets from ports which attached to it. You > >> can't use 2 separate l3fwd > >> to connect 2 VMs. > >> > >> If you are tring to connect 2 VMs, the classic topo is something like: > >> | VM1 | HOST > >> | VM2 | > >> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app > >> > >> J.J. Martzki, > >> Best wishes. > >> > >> Dinesh Kumar 于2023年4月14日周五 01:32写道： > >> > > >> > Hello Experts, > >> > > >> > I am trying to run the following scenario using l3fwd example in > dpdk-stable-21.11.2 release . > >> > > >> > The flow is : > >> > > >> > VM1 > VM2 > >> > ns0->tap0->DPDK <---------------------> DPDK-> bind a > device with IP > >> > > >> > > >> > The first question I have for VM2: > >> > I assigned IP to the device and can see eth and static ip address > via 'ip add ', then after binding it to dpdk, I can not see this > information via ip add because it is not controlled by kernel module. > >> > Is there any way to see ip and ethernet address after binding to DPDK? > >> > Is this process correct to assign IP and then device to DPDK? > >> > > >> > > >> > on VM1: > >> > After starting the DPDK application, I can see a tap0 device created > in kernel mode via 'ip add show'.Then I attached this interface with > network namespace via: > >> > > >> > ip netns add ns0 > >> > ip link set tap0 netns ns0 > >> > ip netns exec ns0 ip addr add dev tap0 > >> > ip netns exec ns0 ip link set tap0 up > >> > ip netns exec ns0 ip link set lo up > >> > > >> > Still, I am not able to ping anything from this ns0. > >> > > >> > Do I need any other configuration to ping via ns0 to VM2? > >> > > >> > Sorry, in case I am asking any dumb questions, I am new to DPDK and > trying to make the use case work. > >> > Any help/pointer is really appreciated. > >> > > >> > >