Use case understanding for l3fwd

Use case understanding for l3fwd

[Dinesh Kumar]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

Hello Experts,

I am trying to run the following scenario using l3fwd example in
dpdk-stable-21.11.2 release .

The flow is :

         VM1
                                 VM2
ns0->tap0<IPV4 IP>->DPDK       <--------------------->  DPDK-> bind a
device  with IP <IPV4 IP >


The first question I have for VM2:
I assigned IP to the device and can see eth and static ip address via  'ip
add ', then after binding it to dpdk, I can not see this information via ip
add because it is not controlled by kernel module.
Is there any way to see ip and ethernet address after binding to DPDK?
Is this process correct to assign IP  and then device to DPDK?


on VM1:
 After starting the DPDK application, I can see a tap0 device created in
kernel mode via 'ip add show'.Then I attached this interface with network
namespace via:

ip netns add ns0
ip link set tap0 netns ns0
ip netns exec ns0 ip addr add <IPV4 Address> dev tap0
ip netns exec ns0 ip link set tap0 up
ip netns exec ns0 ip link set lo up

Still, I am not able to ping anything from this ns0.

Do I need any other configuration to ping via ns0 to VM2?

Sorry, in case I am asking any dumb questions, I am new to DPDK and trying
to make the use case work.
Any help/pointer is really appreciated.

[-- Attachment #2: Type: text/html, Size: 1587 bytes --]

Re: Use case understanding for l3fwd

[J.J. Martzki]

> Is this process correct to assign IP  and then device to DPDK?
No, after binding the driver to vfio/uio, the IP configuration is no
longer available.
The IP is something belonging to an application, not a driver (DPDK).

> Is there any way to see ip and ethernet address after binding to DPDK?
MAC is available for the rte_eth_dev, there's some API like
rte_eth_macaddr_get() to get MAC
from a specific port. l3fwd should have some CLI commands for getting
MAC ethier, maybe
you can read it's documents.
IP is not available as I said before.

BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate
VMs? It's not gonna work.
AFAK, l3fwd switches packets from ports which attached to it. You
can't use 2 separate l3fwd
to connect 2 VMs.

If you are tring to connect 2 VMs, the classic topo is something like:
|       VM1       |                          HOST
     |        VM2       |
app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app

J.J. Martzki,
Best wishes.

Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月14日周五 01:32写道：
>
> Hello Experts,
>
> I am trying to run the following scenario using l3fwd example in dpdk-stable-21.11.2 release .
>
> The flow is :
>
>          VM1                                                                                                 VM2
> ns0->tap0<IPV4 IP>->DPDK       <--------------------->  DPDK-> bind a device  with IP <IPV4 IP >
>
>
> The first question I have for VM2:
> I assigned IP to the device and can see eth and static ip address via  'ip add ', then after binding it to dpdk, I can not see this information via ip add because it is not controlled by kernel module.
> Is there any way to see ip and ethernet address after binding to DPDK?
> Is this process correct to assign IP  and then device to DPDK?
>
>
> on VM1:
>  After starting the DPDK application, I can see a tap0 device created in kernel mode via 'ip add show'.Then I attached this interface with network namespace via:
>
> ip netns add ns0
> ip link set tap0 netns ns0
> ip netns exec ns0 ip addr add <IPV4 Address> dev tap0
> ip netns exec ns0 ip link set tap0 up
> ip netns exec ns0 ip link set lo up
>
> Still, I am not able to ping anything from this ns0.
>
> Do I need any other configuration to ping via ns0 to VM2?
>
> Sorry, in case I am asking any dumb questions, I am new to DPDK and trying to make the use case work.
> Any help/pointer is really appreciated.
>
>

Re: Use case understanding for l3fwd

[J.J. Martzki]

From your picture, l3fwd is running in VM2, with a tap vdev. But
what's the relationship between VM1 and VM2? Are they running on the
same host?

I assume:
1. VM1 and VM2 are running on the same host.
2. ens05 is a net device in VM1.
3. VM1 has its own net device, let's call it ens06, and IP2 is on it.

Based on the assumptions above, you should make sure VM1 and VM2 can
communicate with each other without l3fwd, just through the linux
network stack.
This ensures the link between these 2 VMs is good (ens05 in VM2  and
ens06 in VM1 can reach each other directly). Maybe they are connected
by a linux bridge on the host or something like that.

And you l3fwd should attach to ens05, by vfio or uio. Just use
dpdk-devbind.py to do this.
After this, ens05 is no longer available in the kernel, and l3fwd can
use it. As your command line '-p 0x3' described, l3fwd will use port 0
and port 1.
You can use l3fwd init log to make sure there are 2 ports available
for l3fwd as we expected.

BTW, I suggest you start from test-pmd first. test-pmd can connect
ports ethier. But it supports interactive mode and many command lines.
Such as display of port info like MAC you mentioned before.

> Please share some link that has details descriptions about testing this flow :
I didn't find a suitable article to describe this, maybe you can try:
https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk
In this article, the DPDK application is ovs, just like your l3fwd.
And it can attach many ports not only vhost ports but also physical
ports, vdev ports like tap and so on.

J.J. Martzki,
Best wishes.


Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月17日周一 23:47写道：
>
> Hi JJ,
> Thank you for the clarification. I am new to this so trying to understand how it will work.
>
> Please share some link that has details descriptions about testing this flow :
>
> |       VM1       |                          HOST
>      |        VM2       |
> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
>
> Sorry for my misunderstanding.
>
> here is the correct flow I am working on now  :
> After running the below command from VM2,  I am pinging IP address of VM1 vi network namespace ns0, but not able to see any message in l3fwd and ping is unreachable.
> Thanks for your time and any help will be really appreciated.
>
>
>
> On Fri, Apr 14, 2023 at 10:46 PM J.J. Martzki <mars14850@gmail.com> wrote:
>>
>> > Is this process correct to assign IP  and then device to DPDK?
>> No, after binding the driver to vfio/uio, the IP configuration is no
>> longer available.
>> The IP is something belonging to an application, not a driver (DPDK).
>>
>> > Is there any way to see ip and ethernet address after binding to DPDK?
>> MAC is available for the rte_eth_dev, there's some API like
>> rte_eth_macaddr_get() to get MAC
>> from a specific port. l3fwd should have some CLI commands for getting
>> MAC ethier, maybe
>> you can read it's documents.
>> IP is not available as I said before.
>>
>> BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate
>> VMs? It's not gonna work.
>> AFAK, l3fwd switches packets from ports which attached to it. You
>> can't use 2 separate l3fwd
>> to connect 2 VMs.
>>
>> If you are tring to connect 2 VMs, the classic topo is something like:
>> |       VM1       |                          HOST
>>      |        VM2       |
>> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
>>
>> J.J. Martzki,
>> Best wishes.
>>
>> Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月14日周五 01:32写道：
>> >
>> > Hello Experts,
>> >
>> > I am trying to run the following scenario using l3fwd example in dpdk-stable-21.11.2 release .
>> >
>> > The flow is :
>> >
>> >          VM1                                                                                                 VM2
>> > ns0->tap0<IPV4 IP>->DPDK       <--------------------->  DPDK-> bind a device  with IP <IPV4 IP >
>> >
>> >
>> > The first question I have for VM2:
>> > I assigned IP to the device and can see eth and static ip address via  'ip add ', then after binding it to dpdk, I can not see this information via ip add because it is not controlled by kernel module.
>> > Is there any way to see ip and ethernet address after binding to DPDK?
>> > Is this process correct to assign IP  and then device to DPDK?
>> >
>> >
>> > on VM1:
>> >  After starting the DPDK application, I can see a tap0 device created in kernel mode via 'ip add show'.Then I attached this interface with network namespace via:
>> >
>> > ip netns add ns0
>> > ip link set tap0 netns ns0
>> > ip netns exec ns0 ip addr add <IPV4 Address> dev tap0
>> > ip netns exec ns0 ip link set tap0 up
>> > ip netns exec ns0 ip link set lo up
>> >
>> > Still, I am not able to ping anything from this ns0.
>> >
>> > Do I need any other configuration to ping via ns0 to VM2?
>> >
>> > Sorry, in case I am asking any dumb questions, I am new to DPDK and trying to make the use case work.
>> > Any help/pointer is really appreciated.
>> >
>> >

Re: Use case understanding for l3fwd

[Dinesh Kumar]

[-- Attachment #1: Type: text/plain, Size: 11252 bytes --]

Hi JJ,
Sorry for the late reply.
Here are the answers to your questions.
Still having the same issue.
Nothing is reaching to DPDK application after ping .
I tried to run l2fwd and found the same issue
on VM2 after running l2fwd I have two Mac addresses for port 0
: FA:16:3E:EF:2C:26  and  Port 1, MAC address: FA:16:3E:0D:39:EC

then I added entry in to ARP table to associate IP address with these mac
address on VM2 like :
arp -s x.y.t.z FA:16:3E:EF:2C:26 dev ens3(ens3 is bind with  kernel module)
arp -s  x.y.t.r  FA:16:3E:0D:39:EC dev ens3
After it I tried to ping   x.y.t.z assuming that this message will go to
port 0 RX queue but not see any message in l2fwd .

Here is a log snapshot:
"iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
done:
Port 0, MAC address: FA:16:3E:EF:2C:26

Initializing port 1... iavf_dev_init_vlan(): Failed to update vlan offload
iavf_dev_configure(): configure VLAN failed: -95
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 25
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied.
Rx Burst Bulk Alloc function will be used on port=1, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_start():  >>
iavf_set_rx_function(): request RXDID[22] in Queue[0] is not supported, set
rx_pkt_burst as legacy for all queues
iavf_set_rx_function(): Using avx2 Vector Rx (port 1).
iavf_set_tx_function(): Using avx2 Vector Tx (port 1).
iavf_configure_queues(): RXDID[22] is not supported, request default
RXDID[1] in Queue[0]
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:FA:16:3E:0D:39:EC
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
iavf_dev_rx_queue_start():  >>
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_dev_alarm_handler(): ICR01_ADMINQ is reported
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
done:
Port 1, MAC address: FA:16:3E:0D:39:EC

Skipping disabled port 2

Checking link statusdone
Port 0 Link up at 25 Gbps FDX Autoneg
Port 1 Link up at 25 Gbps FDX Autoneg
L2FWD: entering main loop on lcore 2
L2FWD:  -- lcoreid=2 portid=1
L2FWD: entering main loop on lcore 1
L2FWD:  -- lcoreid=1 portid=0"

and then log I added at RX queue to dump source and dest in Rx buffer:
L2FWD: ITE one ----nb_rx=1 portid=0
L2FWD: ITR ---2----src=134610945 dest=134219268
I added this login
/* main processing loop */
static void
l2fwd_main_loop(void)
{

 in
/* Read packet from RX queues. 8< */  second loop

 for (i = 0; i < qconf->n_rx_port; i++) {
                        portid = qconf->rx_port_list[i];
                        nb_rx = rte_eth_rx_burst(portid, 0,
                                                 pkts_burst, MAX_PKT_BURST);
                        port_statistics[portid].rx += nb_rx;

                        for (j = 0; j < nb_rx; j++) {
                                m = pkts_burst[j];
                                uint32_t ip_dst,ip_src;
                                struct rte_ipv4_hdr *ip_hdr;
                                rte_prefetch0(rte_pktmbuf_mtod(m, void *));
                                ip_hdr = rte_pktmbuf_mtod(m, struct
rte_ipv4_hdr *);
                                ip_dst = rte_be_to_cpu_32(ip_hdr->dst_addr);
                                ip_src = rte_be_to_cpu_32(ip_hdr->src_addr);
                                RTE_LOG(INFO, L2FWD, "ITE one ----nb_rx=%u
portid=%u\n", nb_rx,portid);
                                RTE_LOG(INFO, L2FWD, "ITR ---2----src=%u
dest=%u\n",ip_src,ip_dst);
                                l2fwd_simple_forward(m, portid);
                        }
                }
Please let me know why the ping message is not reaching dpdk even though on
the source side

Thanks,
Dinesh

On Tue, Apr 18, 2023 at 3:51 AM J.J. Martzki <mars14850@gmail.com> wrote:

> From your picture, l3fwd is running in VM2, with a tap vdev. But
> what's the relationship between VM1 and VM2? Are they running on the
> same host?
>
Yes, They are running on the same host with  3 NICs two are bind with dpdk
and one is with kernel module, These NICs are on separate subnets

>
> I assume:
> 1. VM1 and VM2 are running on the same host.
> 2. ens05 is a net device in VM1.
> 3. VM1 has its own net device, let's call it ens06, and IP2 is on it.
> Yes:
>
0000:00:05.0 'Ethernet Virtual Function 700 Series 154c'
drv=uio_pci_generic unused=iavf,vfio-pci
0000:00:06.0 'Ethernet Virtual Function 700 Series 154c'
drv=uio_pci_generic unused=iavf,vfio-pci

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device 1000' if=ens3 drv=virtio-pci
unused=vfio-pci,uio_pci_generic *Active*


> Based on the assumptions above, you should make sure VM1 and VM2 can
> communicate with each other without l3fwd, just through the linux
> network stack.
>
Yes  . I tested with ens05 and ens03 and ping is working fine without dpdk.

> This ensures the link between these 2 VMs is good (ens05 in VM2  and
> ens06 in VM1 can reach each other directly). Maybe they are connected
> by a linux bridge on the host or something like that.
>
> And you l3fwd should attach to ens05, by vfio or uio. Just use
> dpdk-devbind.py to do this.
> After this, ens05 is no longer available in the kernel, and l3fwd can
> use it. As your command line '-p 0x3' described, l3fwd will use port 0
> and port 1.
> You can use l3fwd init log to make sure there are 2 ports available
> for l3fwd as we expected.
>
> BTW, I suggest you start from test-pmd first. test-pmd can connect
> ports ethier. But it supports interactive mode and many command lines.
> Such as display of port info like MAC you mentioned before.
>
> > Please share some link that has details descriptions about testing this
> flow :
> I didn't find a suitable article to describe this, maybe you can try:
>
> https://www.redhat.com/en/blog/how-vhost-user-came-being-virtio-networking-and-dpdk
> In this article, the DPDK application is ovs, just like your l3fwd.
> And it can attach many ports not only vhost ports but also physical
> ports, vdev ports like tap and so on.
>
> J.J. Martzki,
> Best wishes.
>
>
> Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月17日周一 23:47写道：
> >
> > Hi JJ,
> > Thank you for the clarification. I am new to this so trying to
> understand how it will work.
> >
> > Please share some link that has details descriptions about testing this
> flow :
> >
> > |       VM1       |                          HOST
> >      |        VM2       |
> > app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
> >
> > Sorry for my misunderstanding.
> >
> > here is the correct flow I am working on now  :
> > After running the below command from VM2,  I am pinging IP address of
> VM1 vi network namespace ns0, but not able to see any message in l3fwd and
> ping is unreachable.
> > Thanks for your time and any help will be really appreciated.
> >
> >
> >
> > On Fri, Apr 14, 2023 at 10:46 PM J.J. Martzki <mars14850@gmail.com>
> wrote:
> >>
> >> > Is this process correct to assign IP  and then device to DPDK?
> >> No, after binding the driver to vfio/uio, the IP configuration is no
> >> longer available.
> >> The IP is something belonging to an application, not a driver (DPDK).
> >>
> >> > Is there any way to see ip and ethernet address after binding to DPDK?
> >> MAC is available for the rte_eth_dev, there's some API like
> >> rte_eth_macaddr_get() to get MAC
> >> from a specific port. l3fwd should have some CLI commands for getting
> >> MAC ethier, maybe
> >> you can read it's documents.
> >> IP is not available as I said before.
> >>
> >> BTW, your topo is confusing. Are you running 2 l3fwd in 2 separate
> >> VMs? It's not gonna work.
> >> AFAK, l3fwd switches packets from ports which attached to it. You
> >> can't use 2 separate l3fwd
> >> to connect 2 VMs.
> >>
> >> If you are tring to connect 2 VMs, the classic topo is something like:
> >> |       VM1       |                          HOST
> >>      |        VM2       |
> >> app <- device -> qemu <- pmd -> l3fwd <- pmd -> qemu <- device -> app
> >>
> >> J.J. Martzki,
> >> Best wishes.
> >>
> >> Dinesh Kumar <raidinesh@utexas.edu> 于2023年4月14日周五 01:32写道：
> >> >
> >> > Hello Experts,
> >> >
> >> > I am trying to run the following scenario using l3fwd example in
> dpdk-stable-21.11.2 release .
> >> >
> >> > The flow is :
> >> >
> >> >          VM1
>                                        VM2
> >> > ns0->tap0<IPV4 IP>->DPDK       <--------------------->  DPDK-> bind a
> device  with IP <IPV4 IP >
> >> >
> >> >
> >> > The first question I have for VM2:
> >> > I assigned IP to the device and can see eth and static ip address
> via  'ip add ', then after binding it to dpdk, I can not see this
> information via ip add because it is not controlled by kernel module.
> >> > Is there any way to see ip and ethernet address after binding to DPDK?
> >> > Is this process correct to assign IP  and then device to DPDK?
> >> >
> >> >
> >> > on VM1:
> >> >  After starting the DPDK application, I can see a tap0 device created
> in kernel mode via 'ip add show'.Then I attached this interface with
> network namespace via:
> >> >
> >> > ip netns add ns0
> >> > ip link set tap0 netns ns0
> >> > ip netns exec ns0 ip addr add <IPV4 Address> dev tap0
> >> > ip netns exec ns0 ip link set tap0 up
> >> > ip netns exec ns0 ip link set lo up
> >> >
> >> > Still, I am not able to ping anything from this ns0.
> >> >
> >> > Do I need any other configuration to ping via ns0 to VM2?
> >> >
> >> > Sorry, in case I am asking any dumb questions, I am new to DPDK and
> trying to make the use case work.
> >> > Any help/pointer is really appreciated.
> >> >
> >> >
>

[-- Attachment #2: Type: text/html, Size: 14964 bytes --]