[dpdk-dev] Need help to run test_pmd via sr-iov setup

[dpdk-dev] Need help to run test_pmd via sr-iov setup

[zhi ma]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 5752 bytes --]

 

 Need help to run test_pmd via sr-iov setup
my steps:1. turn on intel vt in bios2. set "intel_iommu=on" in kernel boot parameter3. update host grub and reboot4. create VF:    echo 1 > /sys/class/net/eth17/device sriov_numvfs    echo 1 > /sys/class/net/eth18/device sriov_numvfs    cat /sys/class/net/eth17/device sriov_numvfs    1    cat /sys/class/net/eth18/device sriov_numvfs    1 5. set Mac    ip link set eth17 vf 0 mac aa:bb:cc:dd:00:00    ip link set eth18 vf 0 mac aa:bb:cc:dd:01:00 6. rmmod ixgbevf 7. start ubuntu 14.04 vm with KVM virt-manager, add two PCI host device (Virtual function devices) 8. run dpdk app --- test_pmd in vm (two ports connected back to back)    start tx_first    stop    no packets  9. if I run test_pmd inside host machine, it works fine. (lots of sent and received packets, so those two ports connected fine)
 INTEL X520-SR2 both host and guest vm running as ubuntu 14.04
 any thing wrong with my setup? Please help
part of my virus xml    <hostdev mode='subsystem' type='pci' managed='yes'>      <driver name='vfio'/>      <source>        <address domain='0x0000' bus='0x09' slot='0x10' function='0x0'/>      </source>      <alias name='hostdev0'/>      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>      <driver name='vfio'/>      <source>        <address domain='0x0000' bus='0x09' slot='0x10' function='0x1'/>      </source>      <alias name='hostdev1'/>      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>    </hostdev>
  
From maxime.coquelin@redhat.com  Thu Oct 12 17:39:07 2017
Return-Path: <maxime.coquelin@redhat.com>
Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28])
 by dpdk.org (Postfix) with ESMTP id ECD981B3CE
 for <dev@dpdk.org>; Thu, 12 Oct 2017 17:39:05 +0200 (CEST)
Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com
 [10.5.11.14])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.redhat.com (Postfix) with ESMTPS id EFCC57E38F;
 Thu, 12 Oct 2017 15:39:04 +0000 (UTC)
DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com EFCC57E38F
Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com;
 dmarc=none (p=none dis=none) header.from=redhat.com
Authentication-Results: ext-mx04.extmail.prod.ext.phx2.redhat.com;
 spfúil smtp.mailfrom=maxime.coquelin@redhat.com
Received: from localhost.localdomain (ovpn-112-53.ams2.redhat.com
 [10.36.112.53])
 by smtp.corp.redhat.com (Postfix) with ESMTP id 0B21117D5B;
 Thu, 12 Oct 2017 15:39:02 +0000 (UTC)
From: Maxime Coquelin <maxime.coquelin@redhat.com>
To: yliu@fridaylinux.org,
	thomas@monjalon.net,
	dev@dpdk.org
Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
Date: Thu, 12 Oct 2017 17:38:48 +0200
Message-Id: <20171012153850.21837-1-maxime.coquelin@redhat.com>
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
 (mx1.redhat.com [10.5.110.28]); Thu, 12 Oct 2017 15:39:05 +0000 (UTC)
Subject: [dpdk-dev] [PATCH 0/2] vhost: IOTLB fixes for -rc1
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Oct 2017 15:39:07 -0000

These two patches fixes issues faced when running the VM on a different
socket than DPDK.

In this case, the numa_realloc() function is called to reallocate
the virtqueue and the virtio-net device structs on the VM's socket.

The problem is that doing so corrupts the IOTLB cache list, as the list
head is being reallocated, but the first entry in the list is not updated
to point to the new list head. It results in all new IOTLB entries that
need to be inserted before the first entry in the list to be leaked, as
the new head is still pointing to the first entry at the time the realloc
happened.

Patch 2 addresses this issue by re-initializing the IOTLB cache
completely. Doing this also create again the IOTLB mempool on the new
socket.

This first issue helped to highlight a deadlock that patch 1 fixes.
As inserting an entry before the first entry in the list resulted in a
leak, it ended up flooding Qemu with IOTLB misses for the same address.

The deadlock happen because an optimization was done to lock the iotlb
cache lock once per packet burst instead of once per translation. It
means that when an IOTLB miss is sent, it is done with the lock held.

The problem is that sending an IOTLB miss can block if the socket buffer
is full, and this buffer is emptied by the same Qemu thread which is
waiting for an IOTLB update to be completed. But it never completes
because DPDK waits for the iotlb lock to insert the update into the
iotlb cache, hence the deadlock.

The fix consists in just unlocking the iotlb lock while sending the
IOTLB miss, which is safe as it does not access the iotlb list the
lock protects.

Maxime Coquelin (2):
  vhost: fix deadlock on IOTLB miss
  vhost: fix IOTLB on NUMA realloc

 lib/librte_vhost/iotlb.c      |  1 -
 lib/librte_vhost/vhost.c      | 12 ++++++++++++
 lib/librte_vhost/vhost_user.c |  3 +++
 3 files changed, 15 insertions(+), 1 deletion(-)

--
2.13.6