From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f52.google.com (mail-oi0-f52.google.com [209.85.218.52]) by dpdk.org (Postfix) with ESMTP id D4F278E7B for ; Fri, 11 Dec 2015 15:46:43 +0100 (CET) Received: by oixx65 with SMTP id x65so63751327oix.0 for ; Fri, 11 Dec 2015 06:46:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6wind-com.20150623.gappssmtp.com; s=20150623; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=oHgc/yJG5S3E/6PZbSogzzWd1M0AiV+CsWGSWZgbsUU=; b=UnGRfsPxCEP3tMRgXdUuUhJ7Z4qdtRai+nlqtcuzACSG0HV/7MuIjxMb56qFyUggAM G8koxJvQMNHL3796oSZIQtF2Hfan9aPUXrxSE5o2RirrE+VJhKZBUfDBywncXH8RTMs9 VvFK2+nSWLx0iBGJGf2NmNLpBWjutUB0Ecrt+6JB1WTK2ArUefrY1JSC5kCBLbW9+fDO jORVd8AYMe63A6TRQZ+D1EahcqQby0j8RkDCIFAPTF3LkHr7RBltsys49cmLEZlYUZj4 V9FAVe9V0WpdkspxcctClIz23ufXBZpwtIU76jkTuh7XVLwYSBIyNZYMZek01YpkwO+o u7vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc :content-type; bh=oHgc/yJG5S3E/6PZbSogzzWd1M0AiV+CsWGSWZgbsUU=; b=Or/DwQKwfC/ggQflO/qLJT5pJo1quPdGs3J/Md4n0PQEJiAheTm8LrhSXKu6R2AndG 9L96CUJQRKVKhikTvhET3if1ygiFFp65xJM8XKypUMAOdFwy3jHNHNHAx5lCjTXG0+PD Fy3Qpu8MFca5I330YSO4xbTmbpOAf2qJizhgjixIp3BJQwqfxWCkyG5GV2ckTQuRjvTH +Y7qiYOMT9dJHp05ujDmq9NBSFLdGnMEP2F7lpEsRyWjJ6C1i07mnk07+3TdSU2HDd8z 6b9Ta2CVpyMdY600fqHQPr8M29+SDZS9wSoubQRMHCaxyNeWY/fjJd2l9vsTW6fmptLs FY8w== X-Gm-Message-State: ALoCoQnAZGlYgLI8KFyeasbPvFKV3Ak0mfmPVTCsQnt5QYoK9t+1iI47bf25dJ/bBMtdz3Lf0TpKAvCPJ5bEw5QwwzdWfM+BXiGv4RNmBq82LM3nOu0wPGU= MIME-Version: 1.0 X-Received: by 10.202.176.66 with SMTP id z63mr13992789oie.61.1449845203304; Fri, 11 Dec 2015 06:46:43 -0800 (PST) Received: by 10.76.153.5 with HTTP; Fri, 11 Dec 2015 06:46:43 -0800 (PST) Date: Fri, 11 Dec 2015 15:46:43 +0100 Message-ID: From: David Marchand To: "Zhang, Helin" , "Ananyev, Konstantin" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: [dpdk-dev] ixgbevf does not recover from pf reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Dec 2015 14:46:44 -0000 Hello Helin, Konstantin, I noticed this issue quite some time ago (maybe around dpdk-1.3.0, not sure) but had no time to investigate/report. I hit it again with dpdk-2.2.0-rc2, so maybe all dpdk versions are impacted. The test is quite simple : - send icmp packets from a system (tn) to itself using two 10G interfaces connected to two VF ports of a dual port 82599 nic, - run testpmd in a vm owning those two VF ports, - reset a pf interface in the host, - then no connectivity on the associated VF port unless it is completely restarted. In my previous test, I think I also unplugged the fiber then replugged (without touching the pf interface status on the host) and observed the same issue. I looked at ixgbevf pmd, but I can't find a place where the PF asking for reset would be handled except at devinit. Here is the full description of my test : The system owning the two VF ports is a ubuntu 14.04 virtual machine (3.13.0-71-generic kernel). The host running this virtual machine is running ubuntu 14.04 as well (but kernel is newer 3.19.0-39-generic). - host setup : echo 3072 >/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages echo 3072 >/sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages mkdir -p /mnt/huge mount -t hugetlbfs none /mnt/huge echo 1 > "/sys/bus/pci/devices/0000:83:00.0/sriov_numvfs" bridge link set dev ixgbe0 hwmode veb ip link set dev ixgbe0 vf 0 mac 00:09:c0:0e:4e:64 ip link set dev ixgbe0 vf 0 spoofchk off ip link set dev ixgbe0 mtu 9000 ip link set dev ixgbe0 up ip link set dev ixgbe0 address 00:09:c0:0e:4e:64 echo 1 > "/sys/bus/pci/devices/0000:83:00.1/sriov_numvfs" bridge link set dev ixgbe1 hwmode veb ip link set dev ixgbe1 vf 0 mac 00:09:c0:0e:4e:65 ip link set dev ixgbe1 vf 0 spoofchk off ip link set dev ixgbe1 mtu 9000 ip link set dev ixgbe1 up ip link set dev ixgbe1 address 00:09:c0:0e:4e:65 qemu-system-x86_64 -k fr -daemonize -S --enable-kvm -m 3G -mem-path /mnt/huge -cpu host -smp cores=6,threads=2,sockets=1 -serial telnet::53714,server,nowait -serial null -monitor telnet::36576,server,nowait -device virtio-net,mac=DE:AD:DE:01:02:03,netdev=user.0,addr=03 -netdev user,id=user.0 -device pci-assign,host=0000:83:10.0,addr=04 -device pci-assign,host=0000:83:10.1,addr=05 -hda ubuntu-14.04-template.qcow2 -snapshot -vga none -display none - guest setup : echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages mkdir -p /mnt/huge mount -t hugetlbfs none /mnt/huge modprobe uio modprobe igb_uio echo 0000:00:04.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind echo 0000:00:05.0 > /sys/m/bus/pci/drivers/ixgbevf/unbind echo 8086 10ed > /sys/bus/pci/drivers/igb_uio/new_id testpmd -w0000:00:04.0 -w0000:00:05.0 --huge-dir=/mnt/huge -n 4 -c 0xe -- -i [snip] Port 0: 00:09:C0:0E:4E:64 [snip] Port 1: 00:09:C0:0E:4E:65 Checking link statuses... Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed 10000 Mbps - full-duplex Done testpmd> start - tn setup : ip link set dev xaui0 up ip link set dev xaui1 up ip addr add 1.1.1.1/24 dev xaui0 arp -i xaui0 -s 1.1.1.2 00:09:c0:0e:4e:64 - From here, on tn, send icmp packets with a tcpdump running background on the other iface : ping 1.1.1.2 PING 1.1.1.2 (1.1.1.2): 56 data bytes 03:10:15.003047 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4 (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 1 03:10:16.003151 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4 (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 2 03:10:17.003246 00:02:bb:a8:0d:10 > 00:09:c0:0e:4e:64, ethertype IPv4 (0x0800), length 98: IP 1.1.1.1 > 1.1.1.2: icmp 64: echo request seq 3 - Ok, now, kick ixgbe0 pf interface down / up on host system. ip link set dev ixgbe0 down ip link set dev ixgbe0 up - On tn, no packet is received anymore, the only workaround I found is to stop and restart the port 0 : testpmd> stop [snip] Done. testpmd> port stop all Stopping ports... Checking link statuses... Port 0 Link Down Port 1 Link Up - speed 10000 Mbps - full-duplex Done Funny thing here, the link is reported down, while the pf is already back up. testpmd> port start all PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst size no less than 4 (port=0). Port 0: 00:09:C0:0E:4E:64 PMD: ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst size no less than 4 (port=1). Port 1: 00:09:C0:0E:4E:65 Checking link statuses... Port 0 Link Up - speed 10000 Mbps - full-duplex Port 1 Link Up - speed 10000 Mbps - full-duplex Done Thanks. -- David Marchand