[dpdk-users] Host not sharing hugepage with Guest

[dpdk-users] Host not sharing hugepage with Guest

[vkrishnabhat k]

Hi Team,

I am trying to achieve Phy-VM-Phy
(https://ovs-istokes.readthedocs.io/en/stable/howto/dpdk.html) setup using
KVM and OVS-DPDK. I have connected Guest VM to OVS bridge using
dpdkvhostuser ports. I have enabled the hugepage sharing between host and
guest. I have set 1G huge page in host using grub command. Host machine has
148 huge pages free but in guest it shows only 1 hugepage as free.

When I run l2fwd application in guest, it consumes available 1 free
hugepage and if I try to stop the application and check the free available
hugepages, then there is no free hugepages available. Also I am not able to
assign the hugepages in guest using dpdk-setup.sh script.

If I update the hugepages in the host and restart the Guest then Guest
shows 1 huge page free and available.

Can anyone help me to understand this behavior ? Is it the issue with qemu
? I am using qemu-kvm 2.11.1 and dpdk 17.11.4. Ovs version is 2.10.9

why l2fwd application doesn't release the hugepages it consumed once
application exits ?
Also how to assign/share more number of hugepages from host to guest?

Please help me to solve this issue. Find the attached qemu config and
domain xml.

Thanks & Regards,
Venkat
-------------- next part --------------
Qemu config
-------------

ps ax |grep qemu
56079 ?        Sl     2:29 qemu-system-x86_64 -enable-kvm -name guest=Guest,debug-threads=on -S 
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-46-Guest/master-key.aes 
-machine pc-i440fx-bionic,accel=kvm,usb=off,vmport=on,dump-guest-core=off -cpu Haswell-noTSX-IBRS,
vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc_adjust=on,ssbd=on,xsaveopt=on,pdpe1gb=on,
abm=on -m 4096 -realtime mlock=off -smp 3,sockets=3,cores=1,threads=1 -object memory-backend-file,id=ram-node0,
prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/46-Guest,share=yes,size=4294967296 -numa node,nodeid=0,cpus=0-2,
memdev=ram-node0 -uuid 6805bcbc-bc0c-4c3d-b175-6b3706533982 -no-user-config -nodefaults -chardev socket,id=charmonitor,
path=/var/lib/libvirt/qemu/domain-46-Guest/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control 
-rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 
-global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,
masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,
addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,
bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/Guest.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,
unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,id=drive-ide0-0-1,readonly=on -device ide-cd,bus=ide.0,unit=1,
drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,
mac=52:54:00:90:34:8f,bus=pci.0,addr=0x3 -chardev socket,id=charnet1,path=/usr/local/var/run/openvswitch/vhost-user0 -netdev vhost-user,
chardev=charnet1,queues=2,id=hostnet1 -device virtio-net-pci,mrg_rxbuf=on,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=00:00:00:00:00:01,
bus=pci.0,addr=0x8 -chardev socket,id=charnet2,path=/usr/local/var/run/openvswitch/vhost-user1 -netdev vhost-user,chardev=charnet2,
queues=2,id=hostnet2 -device virtio-net-pci,mrg_rxbuf=on,mq=on,vectors=6,netdev=hostnet2,id=net2,mac=00:00:00:00:00:02,bus=pci.0,addr=0x9 
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device 
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,
disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,
vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,
cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 -chardev spicevmc,id=charredir1,
name=usbredir -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
56399 pts/1    S+     0:00 grep --color=auto qemu
root@inb3:/usr/src/dpdk-stable-17.11.4/usertools#


Guest domain xml
----------------

<domain type='kvm'>
  <name>Guest</name>
  <uuid>6805bcbc-bc0c-4c3d-b175-6b3706533982</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='on'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='3' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0-2' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/Guest.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:90:34:8f'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:00:00:00:00:01'/>
      <source type='unix' path='/usr/local/var/run/openvswitch/vhost-user0' mode='client'/>
      <model type='virtio'/>
      <driver queues='2'>
        <host mrg_rxbuf='on'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:00:00:00:00:02'/>
      <source type='unix' path='/usr/local/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <driver queues='2'>
        <host mrg_rxbuf='on'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>



Host Huge page info
-------------------

root@inb3:/usr/src/dpdk-stable-17.11.4/usertools# cat /proc/meminfo | grep Huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:     304
HugePages_Free:      298
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
root@inb3:/usr/src/dpdk-stable-17.11.4/usertools#


Guest Huge page info
--------------------

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       1
HugePages_Free:        1
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
root@admin:/home/ubuntu#