Re: Use case understanding for l3fwd

[Dinesh Kumar <raidinesh@utexas.edu>]

[-- Attachment #1: Type: text/plain, Size: 11252 bytes --]

Hi JJ,
Sorry for the late reply.
Here are the answers to your questions.
Still having the same issue.
Nothing is reaching to DPDK application after ping .
I tried to run l2fwd and found the same issue
on VM2 after running l2fwd I have two Mac addresses for port 0
: FA:16:3E:EF:2C:26  and  Port 1, MAC address: FA:16:3E:0D:39:EC

then I added entry in to ARP table to associate IP address with these mac
address on VM2 like :
arp -s x.y.t.z FA:16:3E:EF:2C:26 dev ens3(ens3 is bind with  kernel module)
arp -s  x.y.t.r  FA:16:3E:0D:39:EC dev ens3
After it I tried to ping   x.y.t.z assuming that this message will go to
port 0 RX queue but not see any message in l2fwd .

Here is a log snapshot:
"iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
done:
Port 0, MAC address: FA:16:3E:EF:2C:26

Initializing port 1... iavf_dev_init_vlan(): Failed to update vlan offload
iavf_dev_configure(): configure VLAN failed: -95
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 25
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied.
Rx Burst Bulk Alloc function will be used on port=1, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_start():  >>
iavf_set_rx_function(): request RXDID[22] in Queue[0] is not supported, set
rx_pkt_burst as legacy for all queues
iavf_set_rx_function(): Using avx2 Vector Rx (port 1).
iavf_set_tx_function(): Using avx2 Vector Tx (port 1).
iavf_configure_queues(): RXDID[22] is not supported, request default
RXDID[1] in Queue[0]
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:FA:16:3E:0D:39:EC
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
iavf_dev_rx_queue_start():  >>
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
done:
Port 1, MAC address: FA:16:3E:0D:39:EC

Skipping disabled port 2

Checking link statusdone
Port 0 Link up at 25 Gbps FDX Autoneg
Port 1 Link up at 25 Gbps FDX Autoneg
L2FWD: entering main loop on lcore 2
L2FWD:  -- lcoreid=2 portid=1
L2FWD: entering main loop on lcore 1
L2FWD:  -- lcoreid=1 portid=0"

and then log I added at RX queue to dump source and dest in Rx buffer:
L2FWD: ITE one ----nb_rx=1 portid=0
L2FWD: ITR ---2----src=134610945 dest=134219268
I added this login
/* main processing loop */
static void
l2fwd_main_loop(void)
{

 in
/* Read packet from RX queues. 8< */  second loop

 for (i = 0; i < qconf->n_rx_port; i++) {
                        portid = qconf->rx_port_list[i];
                        nb_rx = rte_eth_rx_burst(portid, 0,
                                                 pkts_burst, MAX_PKT_BURST);
                        port_statistics[portid].rx += nb_rx;

                        for (j = 0; j < nb_rx; j++) {
                                m = pkts_burst[j];
                                uint32_t ip_dst,ip_src;
                                struct rte_ipv4_hdr *ip_hdr;
                                rte_prefetch0(rte_pktmbuf_mtod(m, void *));
                                ip_hdr = rte_pktmbuf_mtod(m, struct
rte_ipv4_hdr *);
                                ip_dst = rte_be_to_cpu_32(ip_hdr->dst_addr);
                                ip_src = rte_be_to_cpu_32(ip_hdr->src_addr);
                                RTE_LOG(INFO, L2FWD, "ITE one ----nb_rx=%u
portid=%u\n", nb_rx,portid);
                                RTE_LOG(INFO, L2FWD, "ITR ---2----src=%u
dest=%u\n",ip_src,ip_dst);
                                l2fwd_simple_forward(m, portid);
                        }
                }
Please let me know why the ping message is not reaching dpdk even though on
the source side

Thanks,
Dinesh

On Tue, Apr 18, 2023 at 3:51 AM J.J. Martzki <mars14850@gmail.com> wrote:

> From your picture, l3fwd is running in VM2, with a tap vdev. But
> what's the relationship between VM1 and VM2? Are they running on the
> same host?
>
Yes, They are running on the same host with  3 NICs two are bind with dpdk
and one is with kernel module, These NICs are on separate subnets

>
> I assume:
> 1. VM1 and VM2 are running on the same host.
> 2. ens05 is a net device in VM1.
> 3. VM1 has its own net device, let's call it ens06, and IP2 is on it.
> Yes:
>
0000:00:05.0 'Ethernet Virtual Function 700 Series 154c'
drv=uio_pci_generic unused=iavf,vfio-pci
0000:00:06.0 'Ethernet Virtual Function 700 Series 154c'
drv=uio_pci_generic unused=iavf,vfio-pci

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device 1000' if=ens3 drv=virtio-pci
unused=vfio-pci,uio_pci_generic *Active*


> Based on the assumptions above, you should make sure VM1 and VM2 can
> communicate with each other without l3fwd, just through the linux
> network stack.
>
Yes  . I tested with ens05 and ens03 and ping is working fine without dpdk.

> This ensures the link between these 2 VMs is good (ens05 in VM2  and
> ens06 in VM1 can reach each other directly). Maybe they are connected
> by a linux bridge on the host or something like that.
>
> And you l3fwd should attach to ens05, by vfio or uio. Just use
> dpdk-devbind.py to do this.
> After this, ens05 is no longer available in the kernel, and l3fwd can
> use it. As your command line '-p 0x3' described, l3fwd will use port 0
> and port 1.
> You can use l3fwd init log to make sure there are 2 ports available
> for l3fwd as we expected.
>
> BTW, I suggest you start from test-pmd first. test-pmd can connect
> ports ethier. But it supports interactive mode and many command lines.
> Such as display of port info like MAC you mentioned before.
>
> > Please share some link that has details descriptions about testing this
> flow :
> I didn't find a suitable article to describe this, maybe you can try:
>
> https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk
> In this article, the DPDK application is ovs, just like your l3fwd.
> And it can attach many ports not only vhost ports but also physical
> ports, vdev ports like tap and so on.
>
> J.J. Martzki,
> Best wishes.
>
>
> Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月17日周一 23:47写道：
> >
> > Hi JJ,
> > Thank you for the clarification. I am new to this so trying to
> understand how it will work.
> >
> > Please share some link that has details descriptions about testing this
> flow :
> >
> > |       VM1       |                          HOST
> >      |        VM2       |
> > app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
> >
> > Sorry for my misunderstanding.
> >
> > here is the correct flow I am working on now  :
> > After running the below command from VM2,  I am pinging IP address of
> VM1 vi network namespace ns0, but not able to see any message in l3fwd and
> ping is unreachable.
> > Thanks for your time and any help will be really appreciated.
> >
> >
> >
> > On Fri, Apr 14, 2023 at 10:46 PM J.J. Martzki <mars14850@gmail.com>
> wrote:
> >>
> >> > Is this process correct to assign IP  and then device to DPDK?
> >> No, after binding the driver to vfio/uio, the IP configuration is no
> >> longer available.
> >> The IP is something belonging to an application, not a driver (DPDK).
> >>
> >> > Is there any way to see ip and ethernet address after binding to DPDK?
> >> MAC is available for the rte_eth_dev, there's some API like
> >> rte_eth_macaddr_get() to get MAC
> >> from a specific port. l3fwd should have some CLI commands for getting
> >> MAC ethier, maybe
> >> you can read it's documents.
> >> IP is not available as I said before.
> >>
> >> BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate
> >> VMs? It's not gonna work.
> >> AFAK, l3fwd switches packets from ports which attached to it. You
> >> can't use 2 separate l3fwd
> >> to connect 2 VMs.
> >>
> >> If you are tring to connect 2 VMs, the classic topo is something like:
> >> |       VM1       |                          HOST
> >>      |        VM2       |
> >> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
> >>
> >> J.J. Martzki,
> >> Best wishes.
> >>
> >> Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月14日周五 01:32写道：
> >> >
> >> > Hello Experts,
> >> >
> >> > I am trying to run the following scenario using l3fwd example in
> dpdk-stable-21.11.2 release .
> >> >
> >> > The flow is :
> >> >
> >> >          VM1
>                                        VM2
> >> > ns0->tap0<IPV4 IP>->DPDK       <--------------------->  DPDK-> bind a
> device  with IP <IPV4 IP >
> >> >
> >> >
> >> > The first question I have for VM2:
> >> > I assigned IP to the device and can see eth and static ip address
> via  'ip add ', then after binding it to dpdk, I can not see this
> information via ip add because it is not controlled by kernel module.
> >> > Is there any way to see ip and ethernet address after binding to DPDK?
> >> > Is this process correct to assign IP  and then device to DPDK?
> >> >
> >> >
> >> > on VM1:
> >> >  After starting the DPDK application, I can see a tap0 device created
> in kernel mode via 'ip add show'.Then I attached this interface with
> network namespace via:
> >> >
> >> > ip netns add ns0
> >> > ip link set tap0 netns ns0
> >> > ip netns exec ns0 ip addr add <IPV4 Address> dev tap0
> >> > ip netns exec ns0 ip link set tap0 up
> >> > ip netns exec ns0 ip link set lo up
> >> >
> >> > Still, I am not able to ping anything from this ns0.
> >> >
> >> > Do I need any other configuration to ping via ns0 to VM2?
> >> >
> >> > Sorry, in case I am asking any dumb questions, I am new to DPDK and
> trying to make the use case work.
> >> > Any help/pointer is really appreciated.
> >> >
> >> >
>

[-- Attachment #2: Type: text/html, Size: 14964 bytes --]