Re: [dpdk-dev] How to debug packet sends to virtual functions

[jigsaw <jigsaw@gmail.com>]

Hi Mats,

Yes the virsh xml is created in essentially the same way. And the
setup in host is also the same as yours.
But no PCI passthru. As I understand, in a PF-VF deployment it makes
no sense for PCI passthru, coz the PCI device must be visible and
under control of host. (Pls correct me if Im wrong).

-Qinglai

On Tue, Feb 4, 2014 at 1:31 PM, Mats Liljegren
<liljegren.mats2@gmail.com> wrote:
> Hi Qinglai,
>
> How did you configure the PCI passthrough in KVM? Did you use command
> line parameter as described in the instructions, or do you use virsh
> XML files, or maybe using virt-manager?
>
> The steps that works best (i.e. can receive but not transmit) are:
>
> 1) sudo modprobe ixgbe max_vfs=2
> 2) sudo ip link set em1 vf 0 mac <phys mac>
> 3) sudo ifconfig em1 up
> 4) sudo modprobe pci-stub
> 5) echo "8086 1515" | sudo tee /sys/bus/pci/drivers/pci-stub/new_id
> 6) echo "0000:02:10.0" | sudo tee
> /sys/bus/pci/devices/0000\:02\:10.0/driver/unbind
> 7) echo "0000:02:10.0" | sudo tee /sys/bus/pci/drivers/pci-stub/bind
> 8) virsh create vm vm.xml
>
> The guest then does:
>
> 5) sudo modprobe uio
> 6) sudo insmod $RTE_SDK/$RTE_TARGET/kmod/igb_uio 0000:02:10.0
> 7) sudo $RTE_SDK/tools/pci_unbind.py -b igb_uio
> 8) sudo <run DPDK app>
>
> The virsh xml file has the following entry for the virtual function:
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>     <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x0' />
>       </source>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </hostdev>
>
> I actually have two virtual functions and two physical functions, but
> I just repeat above for each.
>
> Regards
> Mats
>
> On Tue, Feb 4, 2014 at 12:14 PM, jigsaw <jigsaw@gmail.com> wrote:
>> Hi Mats,
>>
>> I didn't have any deviation. What I did is just loading ixgbe (with
>> extra params for vf, as you mentioned in your first email),
>> and DPDK is up and running in guest. Of course I also followed section
>> 8.10 of the DPDK release notes.
>>
>> I can switch between DPDK and ixgebvf in guest at runtime and
>> everything works fine.
>>
>> Sorry I can't help to debug coz I have only 82599EB at hand.
>>
>> -Qinglai
>>
>> On Tue, Feb 4, 2014 at 12:45 PM, Mats Liljegren
>> <liljegren.mats2@gmail.com> wrote:
>>> Hi Qinglai,
>>>
>>> Thanks for the response!
>>>
>>> My previous attempt was with ixgbe loaded in the host. I also needed
>>> to load ixgbevf, but this seems to be because of a short-coming in
>>> libvirt. Maybe loading ixgbevf and then unbind it when running the
>>> guest is what causes my problems. I could receive packets with this
>>> setup, but not transmit. I used extra debug to syslog, and it showed
>>> that packets was placed in the transmit queue, but these packets was
>>> never sent.
>>>
>>> I'll see if I can get this working without loading ixgbevf in the host.
>>>
>>> What instructions did you follow to get this working? Did you do any
>>> deviation from the instructions?
>>>
>>> Regards
>>> Mats
>>>
>>> On Tue, Feb 4, 2014 at 11:26 AM, jigsaw <jigsaw@gmail.com> wrote:
>>>> Hi Mats,
>>>>
>>>> I've tried vf with 82599EB and it works fine. As long as the VF is
>>>> visible in guest, DPDK's VF driver should work just as ixgbevf, which
>>>> shares more or less the same code.
>>>>
>>>> I don't understand why you would expect DPDK@guest to work as VF,
>>>> while the host has no ixgbe loaded.
>>>> To make further debug I'd suggest compile ixgbe driver with your own
>>>> syslogs. At least you will be able to see the signalling between vf
>>>> and ixgbe drivers.
>>>>
>>>> -Qinglai
>>>>
>>>>
>>>> On Tue, Feb 4, 2014 at 12:08 PM, Mats Liljegren
>>>> <liljegren.mats2@gmail.com> wrote:
>>>>> This is my fourth mail in my desperate attempt to get DPDK running in
>>>>> KVM and no comments so far, not even any "it works for me". Am I the
>>>>> only one crazy enough to believe that this can be done?
>>>>>
>>>>> Anyway, out of desperation I tried to get it running without having
>>>>> ixgbe or ixgbevf kernel modules loaded in host nor guest. I followed
>>>>> the instructions in the Programmer's Guide, chapter "Setting Up a KVM
>>>>> Virtual Machine Monitor", using the PMD version of the instructions.
>>>>>
>>>>> I get as far as being able to see my four virtual functions in the
>>>>> guest using "lspci". But starting the DPDK application gives me the
>>>>> following error:
>>>>>
>>>>> PMD:    The MAC address is not valid.
>>>>>         The most likely cause of this error is that the VM host
>>>>>         has not assigned a valid MAC address to this VF device.
>>>>>         Please consult the DPDK Release Notes (FAQ section) for
>>>>>         a possible solution to this problem.
>>>>>
>>>>> This may be true, but without any kernel modules loaded, how am I
>>>>> supposed to change any MAC addresses? Can this be done from within
>>>>> DPDK?
>>>>>
>>>>> As a side-note, I did try to load ixgbevf in the guest, but it
>>>>> produced no interfaces. There was no error messages in the syslog
>>>>> though.
>>>>>
>>>>> Is it possible to get X540 working in a guest or should I switch hardware?
>>>>>
>>>>> Since the instructions assumes I know the command line to KVM to start
>>>>> my guest (which I do not), I cannot followed them precisely. I use
>>>>> virsh and XML file, and maybe I've misunderstood how to translate the
>>>>> pci-assign parameter to XML code. I currently use a <hostdev> entry,
>>>>> but I've also tried <interface type='hostdev'>. Neither has been
>>>>> working for me so far, though the <interface> version got me as far as
>>>>> being able to receive packets at least, but not transmitting.
>>>>>
>>>>> Regards
>>>>> Mats
>>>>>
>>>>>
>>>>> On Mon, Feb 3, 2014 at 12:13 PM, Mats Liljegren
>>>>> <liljegren.mats2@gmail.com> wrote:
>>>>>> Never mind, I was hit by the infamous MAC spoofing... I got it working
>>>>>> on both the host and the guest using ixgbevf driver, so apparently the
>>>>>> cables are correctly attached.
>>>>>>
>>>>>> Using DPDK is still no-go. It can receive packets, but when sending
>>>>>> the packets the function returns success, but the driver reports
>>>>>> nothing (i.e. no errors, no sent packets, no nothing, except for
>>>>>> received packets of course).
>>>>>>
>>>>>> What could cause this behavior?
>>>>>>
>>>>>> Regards
>>>>>> Mats
>>>>>>
>>>>>> On Fri, Jan 31, 2014 at 7:30 PM, Mats Liljegren
>>>>>> <liljegren.mats2@gmail.com> wrote:
>>>>>>> I have a follow-up on this:
>>>>>>>
>>>>>>> ixgbe version 3.13.10-k
>>>>>>> ixgbevf version 2.7.12-k
>>>>>>>
>>>>>>> (These are what was provided by Ubuntu 13.10)
>>>>>>>
>>>>>>> I tried the following sequence on the host, before starting the guest:
>>>>>>> 1) sudo rmmod ixgbe
>>>>>>> 2) sudo modprobe ixgbe max_vfs=2
>>>>>>> 3) sudo ifconfig em1 up  # This is the physical function
>>>>>>> 4) sudo ifconfig em1_0 192.168.2.2  # This is the virtual function
>>>>>>> 5) ping 192.168.2.1
>>>>>>>
>>>>>>> I can see that the ping request reaches its target, and a reply is
>>>>>>> sent back. But this reply is not received by the ping shell command.
>>>>>>>
>>>>>>> Why?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mats
>>>>>>>
>>>>>>> On Wed, Jan 29, 2014 at 6:56 PM, Mats Liljegren
>>>>>>> <liljegren.mats2@gmail.com> wrote:
>>>>>>>> I'm trying to get a modified version of the l2fwd example running, and
>>>>>>>> have problems with packets being silently thrown away. I can receive
>>>>>>>> packets, and my printf's indicates that the packets are being sent to
>>>>>>>> the correct port, using correct MAC address as source address. And
>>>>>>>> still, the packets are lost.
>>>>>>>>
>>>>>>>> Since the port is a virtual function, it seems like I cannot use
>>>>>>>> tcpdump on it to see the network traffic. There is nothing coming out
>>>>>>>> of the cable (activity light not flashing, the receiving end running
>>>>>>>> tcpdump does not see any traffic).
>>>>>>>>
>>>>>>>> I'm using two X540 with two virtual functions each. The application
>>>>>>>> runs in a KVM/Qemu environmen.
>>>>>>>>
>>>>>>>> Any suggestions how to debug this?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mats