From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <ranlfg@gmail.com>
Received: from mail-lf0-f52.google.com (mail-lf0-f52.google.com
 [209.85.215.52]) by dpdk.org (Postfix) with ESMTP id 4CA732C1A
 for <users@dpdk.org>; Wed, 12 Jul 2017 15:30:10 +0200 (CEST)
Received: by mail-lf0-f52.google.com with SMTP id t72so16645971lff.1
 for <users@dpdk.org>; Wed, 12 Jul 2017 06:30:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc; bh=+pM87h08rplnI4Vg1r/23n6EH7yi21D+WwfRJTAz170=;
 b=eyr/J1ADWkaGItpWVkHKJ4NcHDYBT1rTHqbj1815AMesbpTKVGXHjmmG62qJn8SJuB
 elQoib2Bb2a20FnPTvKFFclEXlE9vgonfGsEbtPoBwXnKdC8gC7wYHKiB8VY/Eq1d/1f
 yGiAdHX8OHZrIb/fSBlo+ga0g1w65wAkzd2vcZD0SsXRCp7yw+D+KDpybC7UDuHWSNmd
 HF1FPjXBnZlheo2AKMNnsUmMlrZqN4MGSl7NXN/MziFaSG3rY8GhpQNUly2IeInJspAV
 1UeufE4bA6DgC0K03GoZ6XYNWd83bmrNoXEHd2vflKh/ESC3v7dPPESNn/3Amtl+5+aG
 NamA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20161025;
 h=x-gm-message-state:mime-version:in-reply-to:references:from:date
 :message-id:subject:to:cc;
 bh=+pM87h08rplnI4Vg1r/23n6EH7yi21D+WwfRJTAz170=;
 b=MC48CzdZC8/CPR5fpCUb06M0bG/VcKWSU8glMoS1SnroyVetKKuZPl8NMCe0s37RXU
 fUSHJLQ8gXh25DvyWgWLDt34ZL8mGKm6R7jR/JNVw8d+dagHKBl7gnjAIUgW+jGt2N0q
 5L2oZpgYcvn0GnxVTI+vfJlCfQoaeo7s5isGjQb8+R76LIC3aitMiIrYSnMboByrQ0bJ
 SmFRk0+cMWuU1AXGdQ9VX1H2qm3bEDzOzfog/89GrKOb2F/FOx/0o1L+v+tjPb9ZqCPc
 M7TbYiqUb2GIyBpkOObpiU1oqjMu6N5ANH4hCZpFMcaa5U/xm8HS7nlnEBZWTtWzyN7z
 1utg==
X-Gm-Message-State: AIVw110bFAZHfm7ma31OLhdRfETZ4FOFppG+F9ECWzmUHZpPukwv/aAP
 M+Xb4jczLE3N78rz5GVDehK+63nNqw==
X-Received: by 10.46.70.26 with SMTP id t26mr1799848lja.7.1499866209623; Wed,
 12 Jul 2017 06:30:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.25.195.10 with HTTP; Wed, 12 Jul 2017 06:30:08 -0700 (PDT)
In-Reply-To: <9B0331B6EBBD0E4684FBFAEDA55776F93DA45094@HASMSX110.ger.corp.intel.com>
References: <CAPaJkxPf-=gtcuTcQxk4XTatdQYoykoEJqoEu_aH9H+6Ff=KiA@mail.gmail.com>
 <9B0331B6EBBD0E4684FBFAEDA55776F93DA45021@HASMSX110.ger.corp.intel.com>
 <CAPaJkxOpn3CaiUkdep_6Mq1wgkV-js=teVfbsQsoo+XQZxvQWw@mail.gmail.com>
 <9B0331B6EBBD0E4684FBFAEDA55776F93DA45094@HASMSX110.ger.corp.intel.com>
From: ran lifshitz <ranlfg@gmail.com>
Date: Wed, 12 Jul 2017 16:30:08 +0300
Message-ID: <CAPaJkxNPOBLYc5Ec_6gSu_XURBAxYGCJAXTd4ySW3Y5ebngwOQ@mail.gmail.com>
To: "Rosen, Rami" <rami.rosen@intel.com>
Cc: "users@dpdk.org" <users@dpdk.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: Re: [dpdk-users] Using PF bound to IGB_UIO,
 together with sr-iov VFs passed to guests
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Jul 2017 13:30:10 -0000

Yeah I assume something is going wrong with libvirt logic, direct qemu
command is probably working.
>>From the logs it might be related to udev interface which does not detect
properly the vf when libvirt tries to discover the interface .
Can you repeat your scenario, but with libvirt ?


No specific kernel logs from kernel.


here is libvirt debug info:

2000-01-18 00:10:51.573+0000: 2055: info : virEventPollRunOnce:641 :
EVENT_POLL_RUN: nhandles=3D8 timeout=3D4979
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollRunOnce:651 : Poll
got 1 event(s)
2000-01-18 00:10:51.574+0000: 2055: debug :
virEventPollDispatchTimeouts:433 : Dispatch 2
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:479
: Dispatch 8
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D0 w=3D1
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D1 w=3D2
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D2 w=3D3
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D3 w=3D4
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D4 w=3D5
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D5 w=3D6
2000-01-18 00:10:51.574+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D6 w=3D7
2000-01-18 00:10:51.574+0000: 2055: info : virEventPollDispatchHandles:507
: EVENT_POLL_DISPATCH_HANDLE: watch=3D7 events=3D1
2000-01-18 00:10:51.574+0000: 2055: debug : udevEventHandleCallback:1543 :
udev action: 'add'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:124 :
udev reports device '35' does not have property 'DRIVER'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceProperty:139 :
Found property key 'SUBSYSTEM' value 'vfio' for device with sysname '35'
2000-01-18 00:10:51.574+0000: 2055: debug : udevGetDeviceType:1245 : Could
not determine device type for device with sysfs name '35'
2000-01-18 00:10:51.574+0000: 2055: debug : udevAddOneDevice:1413 :
Discarding device -1 0x5641bbfd2570 /sys/devices/virtual/vfio/35

and later i see:


2000-01-18 00:10:51.575+0000: 2055: info : virEventPollDispatchHandles:507
: EVENT_POLL_DISPATCH_HANDLE: watch=3D7 events=3D1
2000-01-18 00:10:51.575+0000: 2055: debug : udevEventHandleCallback:1543 :
udev action: 'remove'
2000-01-18 00:10:51.575+0000: 2055: debug : udevRemoveOneDevice:1314 :
Failed to find device to remove that has udev name
'/sys/devices/virtual/vfio/35'
2000-01-18 00:10:51.575+0000: 2055: debug : virEventPollDispatchHandles:493
: i=3D7 w=3D31


which causes the vm bootup to fail:
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerProgramSendError:153 : prog=3D536903814 ver=3D1 proc=3D10 type=
=3D1
serial=3D5 msg=3D0x5641bbfd3b90 rerr=3D0x7f49bc0b7db0
2000-01-18 00:10:51.576+0000: 2057: debug : virNetMessageEncodePayload:376
: Encode length as 120
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientSendMessageLocked:1397 : msg=3D0x5641bbfd3b90 proc=3D10
len=3D120 offset=3D0
2000-01-18 00:10:51.576+0000: 2057: info :
virNetServerClientSendMessageLocked:1405 : RPC_SERVER_CLIENT_MSG_TX_QUEUE:
client=3D0x5641bbfd59b0 len=3D120 prog=3D536903814 vers=3D1 proc=3D10 type=
=3D1 status=3D1
serial=3D5
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientCalculateHandleMode:157 : tls=3D(nil) hs=3D-1,
rx=3D0x5641bbfddc30 tx=3D0x5641bbfd3b90
2000-01-18 00:10:51.576+0000: 2057: debug :
virNetServerClientCalculateHandleMode:192 : mode=3D3



the xml file:
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made
using:
virsh edit vm64efi
or other application using the libvirt API.
-->
<!--  domain type=3D'qemu'  -->
<domain xmlns:qemu=3D"http://libvirt.org/schemas/domain/qemu/1.0" type=3D"k=
vm">
<name>VoidVNF-SRIOV</name>
<uuid>05ed74d9-e31f-80c2-fb69-6cb052abdb6b</uuid>
<memory unit=3D"M">3072</memory>
<currentMemory unit=3D"M">3072</currentMemory>
<qemu:commandline>
<qemu:arg value=3D"-object"/>
<qemu:arg
value=3D"memory-backend-file,id=3Dmem,size=3D3072M,mem-path=3D/mnt/hugepage=
s,share=3Don"/>
<qemu:arg value=3D"-numa"/>
<qemu:arg value=3D"node,memdev=3Dmem"/>
<qemu:arg value=3D"-mem-prealloc"/>
</qemu:commandline>
<vcpu placement=3D"static">2</vcpu>
<os>
<type>hvm</type>
<loader>/mnt/vm_snapshot/OVMF.fd</loader>
<boot dev=3D"hd"/>
</os>

<qemu:commandline>
<qemu:arg value=3D'-cpu'/>
<qemu:arg value=3D'host'/>
</qemu:commandline>

<vcpu placement=3D'static' cpuset=3D'0'>2</vcpu>
<cputune>
<vcpupin vcpu=3D'0' cpuset=3D'12'/>
<vcpupin vcpu=3D'1' cpuset=3D'13'/>
</cputune>

<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset=3D"utc"/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<!-- on_crash>restart</on_crash -->
<devices>
<!--  emulator>/usr/bin/kvm-spice</emulator  -->
<disk type=3D"file" device=3D"disk">
<driver name=3D"qemu" type=3D"qcow2"/>
<source file=3D"/mnt/vm_snapshot/MiniVoidVNF.qcow2"/>
<!--  target dev=3D'sda' bus=3D'sata'/  -->
<target dev=3D"hda" bus=3D"ide"/>
<address type=3D"drive" controller=3D"0" bus=3D"0" target=3D"0" unit=3D"0"/=
>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x04" function=
=3D"0x0"/>
</disk>
<controller type=3D"usb" index=3D"0">
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x01" function=
=3D"0x2"/>
</controller>
<controller type=3D"pci" index=3D"0" model=3D"pci-root"/>


<interface type=3D"hostdev" managed=3D"yes">
<driver name=3D'vfio'/>
<mac address=3D"52:54:12:34:56:78"/>
<source>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x05" slot=3D"0x10" function=
=3D"0x0"/>
</source>
<parameters interfaceid=3D"0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/>
<vlan>
<tag id=3D"30"/>
</vlan>
</interface>
<interface type=3D"hostdev" managed=3D"yes">
<driver name=3D'vfio'/>
<mac address=3D"52:54:12:34:56:79"/>
<source>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x05" slot=3D"0x10" function=
=3D"0x2"/>
</source>
<parameters interfaceid=3D"0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/>
<vlan>
<tag id=3D"40"/>
</vlan>
</interface>

<!--
interface type=3D'bridge'>
<mac address=3D'52:54:00:85:0e:d0'/>
<source bridge=3D'lxcbr0'/>
<model type=3D'virtio'/>
<address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x03' function=
=3D'0x0'/>
</interface
-->
<!--
interface type=3D'bridge'>
<mac address=3D'52:54:00:00:a7:e0'/>
<source bridge=3D'lxcbr0'/>
<model type=3D'virtio'/>
<address type=3D'pci' domain=3D'0x0000' bus=3D'0x00' slot=3D'0x08' function=
=3D'0x0'/>
</interface
-->
<!--interface type=3D"vhostuser">
<mac address=3D"fa:16:3e:07:88:13"/>
<source type=3D"unix" path=3D"/usr/local/var/run/openvswitch/vhost-0ca8d512=
-ae"
mode=3D"client"/>
<target dev=3D"vhost-0ca8d512-ae"/>
<model type=3D"virtio"/>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x03" function=
=3D"0x0"/>
<parameters interfaceid=3D"0ca8d512-ae6f-4a36-9721-378e9d5e6b7b"/>
</interface>
<interface type=3D"vhostuser">
<mac address=3D"fa:16:3e:07:88:14"/>
<source type=3D"unix" path=3D"/usr/local/var/run/openvswitch/vhost-0ca8d512=
-cc"
mode=3D"client"/>
<target dev=3D"vhost-0ca8d512-cc"/>
<model type=3D"virtio"/>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x08" function=
=3D"0x0"/>
<parameters interfaceid=3D"0ca8d512-cc6f-4a36-9721-378e9d5e6b7b"/>
</interface-->
<serial type=3D"pty">
<target port=3D"0"/>
</serial>
<console type=3D"pty">
<target type=3D"serial" port=3D"0"/>
</console>
<input type=3D"tablet" bus=3D"usb"/>
<input type=3D"mouse" bus=3D"ps2"/>
<input type=3D"keyboard" bus=3D"ps2"/>
<graphics type=3D"vnc" port=3D"-1" autoport=3D"yes" keymap=3D"en-us"/>
<video>
<model type=3D"cirrus" vram=3D"9216" heads=3D"1"/>
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x02" function=
=3D"0x0"/>
</video>
<memballoon model=3D"virtio">
<address type=3D"pci" domain=3D"0x0000" bus=3D"0x00" slot=3D"0x05" function=
=3D"0x0"/>
</memballoon>
</devices>
</domain>



On Wed, Jul 12, 2017 at 2:43 PM, Rosen, Rami <rami.rosen@intel.com> wrote:

> Hi Ran,
>
>
>
> >echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
>
> >I'm getting 63 pfs in lspci,
>
>
>
> You mean VFs and not pfs, I assume.
>
>
>
> I tried something similar but with qemu cli, not virsh, using PCU
> passtrhough
>
> (-device pci-assign=E2=80=A6) and i40e, but using igb_uio. I tried it wit=
h one VF
> and it worked OK.
>
>
>
> Do you see any errors in the kernel log (dmesg)?
>
>
>
>
>
> Regards,
>
> Rami Rosen
>
>
>
> *From:* ran lifshitz [mailto:ranlfg@gmail.com]
> *Sent:* Wednesday, July 12, 2017 14:34
> *To:* Rosen, Rami <rami.rosen@intel.com>
> *Cc:* users@dpdk.org
> *Subject:* Re: [dpdk-users] Using PF bound to IGB_UIO, together with
> sr-iov VFs passed to guests
>
>
>
> Hi Rami,
>
>
>
> I'm creating the vfs, while the pf is bound to igb_uio:
>
> echo 63 > /sys/bus/pci/devices/0000\:05\:00.0/max_vfs
>
>
>
> I'm getting 63 pfs in lspci, but when i try to load the vm:
>
>
>
> virsh create mini1.xml
>
> error: Failed to create domain from mini1.xml
>
> error: An error occurred, but the cause is unknown
>
>
>
>
>
> in the dpdk manual, it's refering to qemu, so I probably assume that the
> libvirt try to initialize the VF mac and vlan, but this is not supported =
by
> the igb_uio ?
>
> it works well when ixgbe.ko is bound.
>
>
>
> On Wed, Jul 12, 2017 at 1:11 PM, Rosen, Rami <rami.rosen@intel.com> wrote=
:
>
> Hi, Ran,
>
> What is the internal error you get when bringing up the
> guest ?
>
> Regards,
> Rami Rosen
>
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of ran lifshitz
> Sent: Tuesday, July 11, 2017 12:24
> To: users@dpdk.org
> Subject: [dpdk-users] Using PF bound to IGB_UIO, together with sr-iov VFs
> passed to guests
>
> Hi,
>
> I'm having trouble trying to build a scenario were the host has the PF
> bound to igb_uio, and working with dpdk application, together with sr-iov
> VFs passthrough to guest machines.
>
> Libvirt is not able to bring up the guest, and exit with internal error.
> I assume that the vlan/mac initialization which is working well, when the
> PF is assigned to ixgbe.k does not work when bound to igb_uio.
>
> Is this an igb_uio issue or libvirt ?
> is there a workaround for this ?
>
> Regards
> Ran
>
>
>
>
>
> --
>
>
>
> [image: Google+]
> <https://plus.google.com/111964218283461713253/posts/p/pub>[image:
> LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]*website
> <http://www.ran-lifshitz.com/> **|**  vCard
> <http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> **|*
> *  email*
>
> [image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/>
>
>
> mobile 972-54-4308444
>
>
>
>
>
>
>



--=20

[image: Google+]
<https://plus.google.com/111964218283461713253/posts/p/pub>[image:
LinkedIn] <http://lnkd.in/eGpd2n>[image: Skype]website
<http://www.ran-lifshitz.com/> |  vCard
<http://www.dynasend.com/signatures/vcard/ranlfg-at-gmail.com.vcf> |  email
[image: Ran Lifshitz, , () -] <http://www.ran-lifshitz.com/>

mobile 972-54-4308444